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ABSTRACT 

In April 1983, a questionnaire on the evaluation of 
remedial mathematics programs wax distributed to all mathematics 
chairpersons of American institutions of higher education. About 650 
(25%) of the institutions responded, yielding information on more 
than 500 remedial programs and 250 evaluations. The survey dealt with 
four general areas of inquiry: (1) the responding institutions and 
their college-level mathematics requirements; (2) the remedial 
mathematics programs; (3) the most rigorous evaluations of these 
programs conducted to date; and (4) reactions to these evaluations. 
Twenty-four major results are listed in the summary. Less than half 
of the -responding colleges require all students to take a cursee in 
college-level mathematics, although most require either a course or 
meeting a distribution requirement that includes mathematics, 
Remedial mathematics programs are offered by 86%. Most of the other 
results concern the content of the programs and their evaluations. 
Recommendations are included, and tables summarize the data. (MNS) 
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SUMMARY OF THE FINDINGS 



In April 1983, a questionnaire on the evaluation of remedial math 
programs (RMP's) was distributed to all math chairpersons of American 
institutions of higher education. About 650 (25%) of the chairpersons (or 
their designees) responded, yielding information on more than 500 remedial 
math programs and 250 RMP evaluations. The survey dealt with four 
general areas of inquiry: (1 ) the responding institutions and their 
college-level math requirements, (2) the RMP's, (3) the most rigorous 
evaluations of their RMP's conducted to date, and (4) reactions to these 
evaluations. 

Major Results 

o Less than half (40%) of responding colleges require all their students 
to take a course in college-level mathematics. However, most 
institutions mandate that at least some studentb either take a math 
cou se or meet a distribution requirement which includes the option of 
college-level mathematics. 

o 86% of all responding colleges offer RMP's. The proportion is even 
higher in public two-year institutions (99%), but somewhat lower in 
private four-year institutions (74%). 

o The typical RMP was established in the mid 1970's, although some 4% 
predate 1960. RMP's tended to be established earliest in public 
two-year colleges, and most recently in private four-year colleges. 

o The most common goal (90%) of remedial math programs is to help 
students prepare for subsequent college-level math courses. Other 
goals include preparation for subsequent science or technical courses 
(49%), and preparation to meet a college, university or system-wide 
standard of competency (39%). 

o The RMP curriculum is generally carved out of topics taught in the 
secondary and primary schools. The most commonly taught content 
areas in RMP's are elementary algebra (87%), arithmetic (76%), 
intermediate algebra (52%), trigonometry (21%), plane geometry (15%), 
and pre-calculus (14%). The RMP's of more selective institutions are 
more likely to cover higher levels of math such as pre-calculus; less 
selective institutions are more likely to deal with lower levels such as 
arithmetic. 

The typical RMP consists of two different courses, although about 10% 
are composed of 5 or more courses. Older programs tend to consist 
of more courses than newer ones. 

Nationally, responding colleges with RMP's judge more than 1 out of 
every 3 freshmen to be in need of RMP courses. At 1/3 of 
responding institutions with RMP's, more than half the freshmen are 
considered to be mathematically unprepared. 



Colleges typical!/ employ several measures of mathematical proficiency 
in developing a student's placement profile. Incoming freshmen are 



most commonly placed into RMP courses on the basis of their scores 
on one or more tests (92%). The instruments employed vary — some 
colleges use locally developed tests (53%), some administer 
commercially distributed placement tests (31%), and others place on 
the basis of an admissions test (44%). In addition, nearly half (48%) 
of colleges take into account the high school records of students in 
assigning them to RMP courses. 

o At nearly half (41%) of responding colleges with RMP's, all students 
judged to be mathematically unprepared are mandated to take RMP 
courses. 

o The median RMP enrollment is 125. Most of the larger programs, with 
enrollments exceeding 1000, are housed in public two-year colleges. 

o About 1 /8 of the responding institutions report that they usually 
offer an insufficient number of RMP sections to accommodate student 
demand; these shortages are most common at public institutions. 

o Nearly half (49%) of responding institutions with RMP's report that 
their programs have been evaluated — a much higher rate than 
reported in a national survey conducted ten years earlier. 

o Institutions at which all students are required to take college-level 
math courses are very likely to have evaluated their RMP f s, as well 
as institutions which use tests for placement, which practice 
mandatory placement, and which maintain placement records. 

o Less than half (40%) of RMP evaluations were carried out as the 
result of requests from outside the program ; only about 1 out of 8 
evaluations resulted from requests from outside the college . 

o Relatively few (9%) RMP evaluations used an outside evaluator. 

o The vast majority (86%) of evaluations were conducted without the 
benefit of special funds allocated for the purpose. Where special 
funding was available, the most common sources were the college 
administration (46%) and governmental units and agencies (37%). 

o RMP evaluations tend to be more product- than process-oriented: 
whereas 88% involved an analysis of the effects of the program on 
student performance, only 51% examined the policies and practices 
followed in the RMP. 

o More than 9 out of every 10 RMP evaluations dealt with the 
effectiveness of RMP courses. Among the other concerns which 
evaluations addressed were: the effectiveness of the RMP placement 
system (74%), the appropriateness of the RMP curriculum (59%), and 
directions in which the RMP should be modified (56%). 

o The typical evaluation employed several distinct kinds of evaluative 
data, most commonly: RMP grades (81%), test scores (73%), grades 
earned subsequently in college-level math courses (72%), student 
opinion of the RMP (49%), college retention rates (44%), RMP faculty 
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opinion of the program (41%), student attitudes toward mathematics 
(27%), grades earned in subsequent science/ technical courses (17%), 
non-RMP faculty opinion of the program (17%), college graduation 
rates (13%), and the judgment of visiting experts (8%). 

The most frequently used evaluation designs were: a comparison of 
students before and after the RMP (58%), a comparison of students 
who completed the RMP with those who had initially been exempted 
(40%), a comparison of students who completed RMP courses with 
those who had needed but not taken them (24%), and a comparison of 
recent RMP completers with previous RMP completers (10%) # Only 7% 
employed a comparison involving national norms, while a mere 3% 
compared students exiting from the local college's RMP with those 
leaving another college's RMP. 

Respondents were generally satisfied with the design (74%) and the 
conduct (81%) of their RMP evaluations, as long as the evaluations 
involved a comparison of student outcomes. However, a number of 
respondents noted reservations about the evaluations, mentioning such 
problems as the subjectivity or superficiality of the data, the lack of 
access to or availability of data, poor cooperation from various 
segments of the college community, design or measurement 
inadequacies, the smallness of the sample, the insufficiency of 
available resources, and deficiencies of the evaluator. 

Most evaluations included suggestions for modifying the RMP — 
suggestions which generally were implemented. These suggestions 
dealt with many aspects of RMP's including: reconsidering placement 
criteria, changing the instructional method, redesigning courses or 
course sequences, modifying the curriculum, altering exit criteria, 
creating a math lab, reconsidering program goals, and adopting new 
curricular materials. 

Only one respondent reported an evaluation which included the 
recommendation that the RMP be continued, suggesting that the 
continuation of other programs was generally taken for granted. 

Respondents advised future evaluators of RMP's to: plan ahead, use 
an outside evaluator, identify program needs, use appropriate 
statistical analyses and designs, get faculty and administrative 
support, seek certain characteristics in the evaluator, and look at 
passing rates in non-remedial courses. 
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INTRODUCTION 



It is the rare program coordinator who, confronted with the task of 
evaluating his or her program, looks on the project with unbridled enthu- 
siasm. For one thing, an evaluation, if taken seriously, can be extremely 
time-consuming, draining badly needed resources from more immediate 
concerns. There is also the nagging suspicion that the study ! s results 
may lack validity, either because of the inappropriateness of the 
questions addressed or the flaws in the procedures followed. Even when 
the recommendations made are clearly valid, there may also be the fear 
that they will never be implemented — that they are simply the result of 
a noble but idle exercise. And finally, coordinators, sensitive to their 
vulnerability, may be wary of exposing the weaknesses of their program* 

Yet it is undeniable that there is intense interest among program 
coordinators on the subject of program evaluation, as evidenced by the 
large turnout typical at professional meetings dealing with the subject. 
Most administrators and faculty seem to want to know how they and their 
programs are doing and would like to find ways to improve them. This is 
particularly true in the area of basic skills instruction, for a number of 
reasons. For one thing, the use of experimental instructional techniques 
in remediation, even though widespread, is still controversial. More 
importantly, the very goals of the programs are sometimes looked upon as 
alien to the mission of the college, drawing special fire to the programs 
and their effectiveness. On some campuses, the importance and size of 
the developmental program may be so enormous that its evaluation ranks 
high at the Office of Institutional Research. Finally, basic skills 
programs are frequently financed, at least in part, by outside funds, the 
granting of which requires some kind of systematic evaluative effort — 
the so-called "pit in the plum." 

For mathematics faculty embarking upon the project of evaluating 
their remedial mathematics program (RMP), defining the parameters of the 
task ahead may be more than a little perplexing. True, such pertinent 
skills as the ability to construct a mathematical model or to conduct 
statistical analyses may well be in their bag of tricks. However, the lack 
of definition in the task itself — the lack of consensus as to what 
constitutes a valid evaluation — may be offensive to the mathematician's 
sensibilities. A torrent of questions must be confronted. To begin with, 
how can one attach a precise and operational meaning to such nebulous 
but pivotal terms as "evaluate" and "good"? What evaluative questions 
need to be raised? Should one take a statistical and empirical 
approach — analyzing data on student achievement — or instead invite a 
blue-ribbon panel of purported experts to assess the program? What 
kinds of measurements if any should be taken, and in comparison to what 
standard should the measurements be judged? 

The questions seem endless. To what extent must the achievement 
of program goals be the focus of the evaluation? Where multiple goals 
exist, how ought they to be prioritized? How much should affective 
factors, such as student or faculty attitudes, be considered, as opposed 
to those factors which are strictly cognitive? Should one take a holistic 
perspective, concluding that the program is good or bad or something in 
between, or a more analytic stance the placement mechanism and the 
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textbook work well but pedagogical strategies need rethinking. Where the 
thrust of the evaluation is a search for program strengths and 
weaknesses, how can one identify appropriate directions for improvement? 
And what about cost-effectiveness: while math faculty will be interested 
in test scores and course grades, administrators may focus more on a 
program's cost than its value. 

With all these debatable points, it is no wonder then that the 
literature gives one the impression of tremendous diversity among past 
RMP evaluations, a diversity that covers a multitude of sins. Several 
efforts have been made to introduce some order to this chaos (e.g., Akst 
and Hecht 1981, Smith and Schavio. n.d.). Wolfs analysis (1979), in 
particular, is valuable for the insight which it provides into common • 
methodological abuses. 

Wolf writes of five methods of program evaluation. In his cosmetic > 
method, the evaluator takes a cursory look at the program, and decides if 
it looks good. In cardia c evaluation, one looks at the program and 
decides in one's heart how well the program is doing (somewhat similar, 
he notes, to the use of subclinical findings in medical research). An 
advocate of th* colloquium school of evaluation goes about his or her job 
assembling a group of people associated with the program to discuss its 
effectiveness until some kind of consensus emerges. In the curricular 
method, a new program is judgpd to be sound to the degree that it 
disturbs any existing programs, arrangements or schedules as little as 
possible. Finally, there is the computation al method, in which it is 
considered desirable to actually gather evidence about how a program 
affects student performance; here a program is judged successful not to 
the degree that the data show evidence that the goals were achieved, but 
rather when two or more analyses of the data yield the same result. 

While most evaluations in the real world no doubt transcend this 
taxonomy, it is also true that those individuals responsible for designing 
and conducting RMP evaluations have not always chosen the most rigorous 
approach. For example, we know of at least one published study which 
dealt with a remedial math program that had been designed to exhibit the 
characteristics advocated by a particular national "expert. 11 The 
evaluation, conducted by the program coordinator himself, simply 
ascertained the extent of overlap between the program's characteristics 
and those advocated by the expert! Then again, there are those 
investigations which have as their objective less the assessing of a 
program's intrinsic merit than whitewashing to protect jobs and to improve 
funding. More broadly, Cross (1979), the country's most distinguished 
researcher in the area of developmental education, describes the history 
of remedial evaluation (at least through the 60's) as riddled with 
methodological flaws, emotionalism and politization. Other observers are 
equally critical (Piesco 1978, Robinson 1950). 

It is not the purpose of this monograph to provide the authors with 
a platform from which to promulgate their own views cn the 
aforementioned methodological issues. Rather, our goal — less ambitious 
but more concrete — is simply to document the extent and nature of the 
evaluative activity being conducted throughout the nation on college 
RMP's, as well as to gather personal reactions to RMP evaluations from 
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those who have gone through the experience. As far as we know, this 
effort is the first of its kind. While the literature contains a number of 
surveys on RMP's (e.g., Jorgensen 1981, Lindberg 1976, Baldwin 1975), 
they have dealt only peripherally with RMP evaluation. 

The present monograph is divided into five chapters. The first 
discusses the methodology which we employed in carrying out our study. 
The second chapter focuses on the context in which the reported 
evaluations took place: here we describe the sample of some 700 
institutions, their college-level graduation requirements in mathematics, 
and most importantly, the RMP's themselves. Even readers not princi- 
pally interested in the subject of program evaluation may find this 
descriptive information of interest. The third chapter focuses on our 
primary concern, namely RMP evaluation — the characteristics of colleges 
whose programs were evaluated and the kinds of evaluative activities 
which were conducted. The fourth deals with evaluation consequences, 
associated problems, and reactions. And, in the final chapter, we 
present some conclusions relating to this investigation. 
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SURVEY METHODOLOGY 
Questionnaire Development and Distribution 



In trying to document the patterns of evaluative strategies employed 
in remedial math programs around the country, we saw from the outset 
that whatever questionnaire we developed would have to deal not only 
with the RMP evaluations themselves but also with the context in which 
they took place. In the end, we developed a questionnaire (Appendix) 
which dealt with four areas of inquiry: 

(1) the responding institutions 

(2) their RMP ! s 

(3) the RMP evaluations, and 

(4) the reactions to the evaluations. 

In drafting the questionnaire, we had to take a stand on a number 
of issues which would, to a considerable extent, shape the results of the 
survey. For example, an issue which surfaced immediately was whether 
to impose precise meanings on such pivotal terms as "remedial 11 and 
"evaluation." In the end, we decided not to. Instead, we chose to 
accord our respondents the widest degree of latitude in interpreting these 
terms, not only because we hoped to gain in breadth of replies what we 
would lose in uniformity, but also because we were genuinely interested 
in what meaning would be attached to the terms. However, when there 
had been more than one evaluation, we did not allow respondents the 
latitude to select which evaluation to comment on. Because we wanted to 
focus on their best evaluative efforts, we requested information about 
each respondent's most thorough evaluation conducted to date. 

A less momentous but equally pragmatic issue that arose was the 
extent to which we should include open-ended items on the questionnaire. 
The advantage of a closed-ended (multiple-choice) format include? a 
greater likelihood of obtaining a higher response rate and of collecting, at 
least superficially, comparable data which are more easily tabulated and 
analyzed. The advantage of an open-ended format includes the possibility 
of yielding a more accurate reflection of the truth, with minimal 
restrictions stemming from the investigators 1 preconceptions. In the end, 
we chose to employ both types of items, adopting the open-ended format 
when we were least confident of developing suitable options. In addition, 
we left room for extra comments by the closed-ended items, and also 
included a request for copies of any available evaluation reports. 

In April 1983, we distributed the questionnaire to each of the 
approximately 2,800 American institutions of higher education admitting 
first-year undergraduate students. Because we knew from previous 
surveys that RMP's are generally housed in mathematics departments, we 
mailed the questionnaires to math chairpersons, requesting that they 
forward their copy to the appropriate party on their campus. 



Sample 



About 650 (25%) of the institutions which were contacted responded. 
In attempting to gauge the extent to which characteristics of our 
responding sample resemble those of the entire surveyed population we 
compared the two groups along three standard dimensions for which 
national data were available — type of institution (level and control) size 
of institution (as measured by undergraduate enrollment), and the 
geographical region in which the institution is located. 

This analysis showed that the institutions in our sample are located 
in regions of the country in roughly the same proportions as the totality 
of colleges nationwide (Table A1), but that there are some differences 
between the sample and population in terms of other variables. Compared 
to the national population, our sample contains something of an 
overrepresentation of public four-year colleges and an 
underrepresentation of private four-year colleges (Table A2). However 
the degree of distortion does not seem serious, and the rankinq of 
institutional categories in terms of both level and control follows the same 
order as in the general population. Consistent with this bias, we also 
noted a discrepancy between sample and population in terms of 
institutional size - our sample contains an underrepresentation of small 
institutions and an overrepresentation of large ones (Table A3). 

Overall, we think that the major trends which the results of our 
survey suggest are likely to hold in the institutional population at larqe 
particularly among those colleges which have an active interest in basic 
skills programs. 

Procedures 

To analyze the responses to our open-ended questfons, we developed 
a coding system and coded each open-ended response twice to ensure 
high reliability. 

We analyzed the data in a variety of ways. First, we computed 
response distributions for each item on the questionnaire, in terms of 
both frequencies and percents. We then performed cross tabulations on 
aM variables which we thought might be related, computing values of the 
chi square statistic in order to gauge the statistical significance of 
differences. In an effort to be consistent with other studies of 
postsecondary institutions, we often partitioned our responding 
institutions by type of institution (level and control). 

Our major findings are presented in the body of the monograph, 
while less important ones are given in the Appendix. 
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BACKGROUND: THE COLLEGES AND THEIR REMEDIAL MATH PROGRAMS 



The College s 

The nature of an institution may well impinge on our major con- 
cerns ~ whether the institution offers a remedial mathematics program, 
whether it chooses to evaluate its RMP, and, if so, what particular 
methodology it selects for the evaluation. In order to gain some insight 
into the nature of our responding institutions, we examined them from the 
point of view of a number of variables including their level (two-year or 
four-year), type of control (public or private), the region in which they 
are iocated, their size, and their college-level math requirements. 

* 

In terms of level and control, our largest category of responding 
institutions is private four-year colleges (33%) , followed by public 
two-year colleges (31%), public four-year colleges (25%), private two-year 
colleges (6%), and other (4%) (Table 1)> Thus, the proportions in our 
sample in the categories of public two-year, public four-year, and private 
four-year colleges are fairly comparable. Our responding institutions are 
distributed throughout the country -- approximately 30% of the 
respondents are from the Great Lakes/ Plains region, with the remainder 
spread fairly evenly among the West/ Southwest (22%), the Southeast (24%) 
and the North/Atlantic (24%) (Table 2). 

Because public institutions tend to be larger than private ones, it is 
not surprising that the sizes of our responding institutions vary 
significantly when grouped by type of institution. Public four-year 
colleges are on the average the largest (their median enrollment is 6,000), 
followed by public two-year colleges (with a median enrollment of 3,550), 
private four-year colleges (1 ,200), and private two-year colleges (400) 
(Table 3). 

We also looked at the non-remedial mathematics curricula of our 
respondents and found that most colleges require college-level math 
courses for at least some of their students — only 4% do not. Aside from 
distribution requirements, almost three-fourths require college-level math 
courses either for all students (40%) or for students in certain majors 
(34%) (Table 4). When we examined requirements for college-level 
mathematics by type of institution , we saw that public two-year 
institutions are least likely to require all students to take college-level 
mathematics -- perhaps because of students in non-transfer, technical 
"tracks. 11 Only 23% of these institutions require all students to take 
college-level math, while 59% of public four-year institutions have such 
requirements. On the other hand, over half of public two-year 
institutions require students in certain majors to take college-level 
mathematics, as compared to less than 30% for other types of institutions 
(Table 5), perhaps another reflection of the system of tracks in two-year 
colleges. 

The Remedial Mathematics Programs 

Before beginning our investigation into the ways in which colleges 
have assessed their remedial math efforts, we need to take a brief look at 
the object of their scrutiny, namely the remedial mathematics programs 
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Table 1 

Percent of Institutions of Various Types 
(n=656) 



Public 2-year 


31% 


Private 2-year 


6% 


Public 4-year 


25% 


Private 4-year 


33% 


Other 


H% 


Total 


100% 



Table 2 

Percent of Institutions in Various Regions 
(n=6i»8) 




Table 3 



Percent of Institutions with Various College 
Enrollments by Type of Institution 







College 


Enrollment 






Up to 
1000 


1001- 
5000 


5000+ 


Median 


Public 2-year 
(n=1 98) 


17% 


46% 


37% 


3550 


Private 2-year 
(n=39) 


87% 


10% 


3% 


400 


Public 4-year 
(n=162) 


7% 


38% 


56% 


6000 


Private 4-year 
(n=217) 


43% 


51% 


6% 


1200 


Total 
(n=616) 


28% 


43% 


29% 


2200 


P(X 2 )=.00* 











A small P(X 2 ), the probability of chi-square, provides evidence that the 
differences between the percents in the table are not likely to be due to 
chance. It is customary in educational research to use either .05 or .01 
as the critical level of significance (Ary et al. 1972). 
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Table 4 



Percent of Institutions with Various 
Student Populations Required 
to Take College-level Math 

(n=541) 



Student Populated Required 




All Students 


40% 


Certain Majors 


34% 


Distribution Requirements/All Students 


11% 


Distribution Requirements/Certain Majors 


5% 


No Students 


4% 


Other 


7% 


Total 


100% 
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Table 5 



Percent of Instilutions with Selected 
Student Populations Required 
to Take College-level Math 
by Type of Institution 



Student Population Required 



Public 2-year 
(n=198) 

Private 2-year 
(n=39) 

Public 4-year 
(n=16?) 

Private 4-ye-ar 
(n=217) 



All 
Stuaents 

23% 



40% 

59% 
43% 



Certain 
Majors 

52% 



25% 
29% 
21% 



J*tal 39% 34% 
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themselves. While our survey in no way purports to provide a 
comprehensive description of RMP's, we did take pains, within space 
limitations, to explore some aspects of RMP's, especially those that we 
thought might impinge on a college's evaluative activities. Thus, the 
questionnaire which we constructed includes items relating to program 
goals as well as to various program practices. 

To begin with, our results indicate that 86% of American colleges 
offer remedial math courses (Table 6), identical to the figure reported in 
a recent national survey (Lederman, Ryzewic and Ribaudo 1983). Among 
public colleges, many of which practice non-selective admissions, virtually 
all (99%) two-year institutions have RMP's, as do 91% of four-year 
colleges. In the private sector, three fourths of four-year colleges offer 
courses in basic mathematics, followed by two-year institutions with only 
71%. Note that, whether an institution is public or private seems to play a 
more important role in determining the likelihood of its offering remedial 
math courses than whether its level is two- or four-y~ar. 

A college's decision to offer math remediation may depend not only 
on characteristics of its student body but also on its mission., 
college-level curriculum, finances, and many other factors. We therefore 
examined the colleges in our sample from a number of perspectives 
besides level and typt-. of control. 

For example, we wondered whether large institutions tend to offer 
remedial math courses more often than small ones; such a trend might be 
the case not only because the student bodies of large institutions, by 
virtue of their size, are more likely to contain some mathematically weak 
students, but also because these institutions are frequently public and 
less selective in their admissions policy (Lederman, Ryzewic and Ribaudo 
1983). In fact, a clear pattern with respect to size does emerge from our 
data: larger institutions (as measured by college enrollment) are more 
likely to offer remedial math courses than smaller ones (Table 7). 
Another predictable result is that colleges which have a college-level 
graduation requirement in mathematics fcr all students are more likely to 
have an RMP than those at which math is not required of any student for 
graduation (Table A4). 

While a small proportion of colleges report that their RMP's go back 
to +he 1950's and even earlier (two responding institutions ir the midwest 
and the south have programs which predate the Civil War!), the typical 
RMP opened its doors in the 1970's (Table 8). it seems clear that the 
growing emphasis on educational access, the proliferation of open 
admissions colleges, and the trend toward lowering high school exit 
standards in the late 60's and early 70's contributed to this growth (Akst 
1985, Adelman 1982). Although the rate of proliferation of RMP's has 
declined in the present decade (perhaps the market is nearly saturated), 
new ones continue to appear, with some 75 in our sample (about 1 in 6) 
founded since 1980. 

The public two-year colleges — many of them open-admissions 
institutions founded in the 1960's — tended to develop RMP's early; in 
fact, some 3/4 of their programs predate the median year of 1975 
(Table 9). By contrast, the private four-year colleges, which did not 
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Table 6 



Percent of Institutions Which Offer Remedial 
Math Courses by Type of Institution 



Public 2-year 99% 
(n=198) 

Private 2-year 71% 
(n=38) 

Public 4-year 91% 
(n=162) 

Private 4-year 74% 
(n=219) 

Total 86% 
(n=617) 

P(X 2 )=.00 
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Table 7 



Percent of Institutions Wh-ich Offer 
Remedial Math Courses by College Enrollment 



College Enrollment 

Up te 1890 76% 
(n=198) 

1001-5000 87% 
(n=210) 

5000+ 95% 
(n=204) 

Total 86% 
(n=612) 

P{X 2 )=.0Q 



-14- 



4 ft 



Table 8 



Percent of RMP's Which Began in 
Various Time Spans 

(n=459) 



The Year the RMP Began 




Before 1960 


4% 


1960-1964 


6% 


1965-1969 


14% 


1970-1974 


25% 


1975-1979 


33% 


1980 and Later 


17% 


Total 


100% 



Table 9 

Percent of RMP's Which Started Before 1975 
by Type of Institution 



Public 2-year 74% 
(n=157) 

Private 2-year 35% 
(n=23) 

Public 4-year 42% 
(n=132) 

Private 4-vear 29% 
(n=129) 

Total 49% 
(n=441) 
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extend a welcome as early or as freely to non-traditional students, 
developed basic math programs rather late, with nearly 3/4 of their RMP's 
under 10 years of age. 

Perhaps the key to understanding the national RMP movement lies in 
developing an appreciation of RMP goals. In our survey, the goal most 
commonly reported is to help students prepare for subsequent math 
courses (Table 10). The implication is that students were and are being 
admitted into college unable to succeed in the regular math curriculum 
without some sort of special preliminary assistance. About half our 
respondents also reported that a major goal of their program is to offer 
students some kind of bridge experience for subsequent science and 
technical courses. However, this goal seems usually to be of secondary 
importance (Table A5) and is generally confined to public two-year 
colleges, with their specialized occupational and technical curricula (Table 
A6). In addition, more than a third of our respondents reported that a 
major RMP goal is to prepare students to reach a proficiency standard. 
No doubt, this extension of the minimal competency trend from high 
schools to colleges represents an effort to bolster the integrity of the 
college degree. A small number of respondents wrote in additional 
program goals: to teach students to enjoy mathematics, to help them in 
their personal growth, to promote math or computational "literacy," to 
satisfy the administration, to help students pass the GED exam, to 
promote their general education, and to prepare them for work or 
everyday life. Furthermore, over half the institutions (59%) indicated 
multiple goals, creating at least the potential for conflict in curricular 
development. For example, there may be constant tension over the 
curriculum of an algebra course which is meant to prepare students to 
pass both a subsequent math course and a competency test, unless the 
demands of the two are identical. 

As a rule, the curriculum of RMP's consists of areas of mathematics 
traditionally covered in the secondary or even primary school (Table 11). 
Thus, the most commonly taught content areas are elementary algebra 
(87%) and arithmetic (76%). Also, half the respondents (52%) cover 
intermediate algebra in their programs, followed by trigonometry (21%), 
and plane geometry (15%). It's worth noting that some 14% of RMP's 
count more advanced material such as pre-calculus as remedial. In 
addition, a small number of respondents wrote in the following curricular 
areas: business and consumer mathematics, the use of calculators, 
overcoming math anxiety, "pre-algebra," solid geometry(J), statistics, 
probability, and analytic geometry. 

As one would expect, there <s a tendency for the more selective 
institutions to cover higher levels of mathematics in their RMP's. For 
example, the proportion of private four-year colleges considering 
pre-calculus to be remedial is more than double the corresponding figure 
for public two-year institutions (Table 12), which instead tend to View 
pre-calculus as part of their regular college curriculum (Fey, Albers and 
Fleming 1981). Conversely, only about half the private four-year 
institutions offer arithmetic in their developmental programs, whereas 
virtually all public two-year colleges include arithmetic in their basic math 
curriculum. 



Table 10 

Percent of RMP's with Various Goals 
(n=553) 



Goal 

To Prepare For: 

Subsequent Math Courses 90% 

Subsequent Science/Technical Courses 49% 

A Standard of Competency 39% 

Other 6% 



Table 11 

Percent of RMP's Which Cover 
Various Curricular Areas 

(n=554) 



Arithmetic 76% 

Elementary Algebra 87% 

Intermediate Algebra 52% 

Trigonometry 21% 

Plane Geometry 15% 

Pre-calculus 14% 

Other 6% 



Table 12 



Percent of RMP's Which Cover Selected Curricular 
Areas by Type of Institution 



Public 2-year 
(n=195) 



Arithmetic 
96% 



Intermediate 
Algebra Pre-calculus 



49% 



10% 



Pr'vate 2-year 
<n=27) 



85% 



37% 



4% 



Public 4-year 
(n=148) 



66% 



57% 



14% 



Private 4-year 
(n=162) 



59% 



54% 



22% 



Total 
(n=532) 



76% 



52% 



14% 



P(X2) 



.00 



.17 



.01 
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RMP f s vary widely in size and organizational complexity. While the 
programs typically consist of two courses, about 10% are composed of 5 or 
more courses (Table 13). In general, the number of courses which make 
up an PMP is related to its age. Programs created after 1980 typically 
consist of one course, those founded before 1960 of three (Table 14). 
Many an RMP no doubt evolved from a complex of support services to a 
single course; in time, this course was seen to cover or to leave out so 
much material that additional courses seemed necessary. In a related 
trend, older RMP's have, in general, enrollments many times the size of 
the johnny-come-latelies (Table A7) . 

In our survey, the median RMP enrollment is 125; however, some of 
the programs (12%) are substantially larger, with enrollments exceeding a 
thousand (Table A8) . Not surprisingly, most of these programs are 
housed in public two-year colleges (Table 15), which tend to admit the 
least prepared students. Thus about half (48%) of responding public 
two-year institutions report that at least 50% of their students are in need 
of remedial math courses, in contrast to only 21% of private four-year 
colleges. 

Many remedial programs consume considerable resources including 
staffing, space, and equipment. Most colleges offer expensive support 
services for their RMP's (e.g., tutoring, CAI, etc.), particularly if a 
large proportion of freshmen are involved (Tables A9 and A10). On the 
other hand, sometimes the resources provided an RMP are inadequate; 
roughly 1/8 of our respondents reported that they usually offer an 
insufficient number of RMP sections to accommodate student demand 
(Table A11). This is particularly a bind when placement is mandatory. 
It is ironic that shortages are most common at public institutions, where 
the commitment of resources to remediation, while perhaps greatest, may 
still not meet the huge demand. 

In assessing the magnitude of the challenge which RMP's address, it 
is sobering to note that nationally, colleges judge more than one out of 
every three freshmen to be in need of RMP courses (Table A12). 
Furthermore, student mathematical deficiencies are particularly common on 
some campuses: at a third of our responding colleges, more than half the 
freshmen are judged in need. While those colleges in which the demand 
for math remediation is greatest tend, not surprisingly, to be public or 
two-year, even in private four-year institutions, more than 1/5 of the 
students are characterized as underprepared (Table A13). 

On what grounds is this characterization of mathematical deficiency 
made? Colleges generally place their students into RMP's on the basis of 
one or more measures of mathematical proficiency (Tables 16 and A14). 
By far the most common tack is to interpret scores on tests which have 
been administered to incoming freshmen (92%). This trend clearly reflects 
the recognition that tests have such advantages as timeliness, 
practicability, and uniformity over other placement measures. The 
particular instrument employed varies — some colleges use a locally 
developed test (53%), some administer a commercially distributed placement 
test (31%), and others rely on a general college admissions test doing 
double duty (44%). 



Table 13 

Percent of RMP's Made Up of Various 
Numbers of Courses 

(n=53?) 



Number of Courses 




1 


38% 


2 


27% 


3 


20% 


H 


7% 


5 


4% 


6 


3% 


7+ 


2% 


Total 


100% 



Median=2 
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Table 14 



Median Number of Courses in RMP's by 
the Year That the RMP Began 



The Year the RMP Began Median 

Before 1960 3 
(n=13) 

1960-1964 3 
(n=27) 

1965-1969 3 
(n=61) 

1970-1974 2 
(n=112) 

1975-1979 ! 
(n=149) 

1980 and Later 1 
(n=77) 

Total 2 
(n=537) 
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Table 15 

Percent of RMP's with Large 
Enrollments (1000 or More) 
by Type of Institution 

(n=:-i18) 



Public 2-year 63% 

Private 2-year 0% 

Public 4-year 34% 

Private 4-year 3% 

Total 10 o% 
P(X2)=.00 
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Table 16 



Percent of Institutions with RMP's Which Use 
Various Types of Placement Information 

(n=549) 

Type of Placement Information 

High School Record 43% 

Test 92% 
Admissions Test 44% 
Local Placement Test 53% 
Commercial Placement Test 31% 

Other 13% 
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The use of one placement measure in no way excludes the use of 
another (Lewenthal 1981). In fact, colleges typically employ more than 
one measure in developing a student's placement profile (Table A15). 

Aside from the aforementioned tests, many colleges (48%) use high 
school records in defining a student's remedial status. Other types of 
placement information used by a small number of respondents are: the 
student's self-evaluation or pleasure, a statewide test, interviews, a 
reading test(!), the results of a summer program, and instructor 
recommendations . 

The issue of whether a college should require its mathematically 
unprepared students to take remedial courses is a complex one involving 
the college's attitude towards its students, curriculum, and resources. 
At 41% of responding institutions (Tables 17 and A16), all students 
judged to be mathematically unprepared are clearly mandated to take RMP 
courses. Elsewhere, mandatory placement is less clear cut varying with 
such factors as the student's major, the RMP course or the degree of 
student unpreparedness. Overwhelmingly (96%), institutions with 
mandatory placement assign students to courses on the basis of tests — 
indicating a preference for uniform, objective criteria. Mandatory 
placement may reflect less a distrust of student judgment than a sense of 
institutional commitment to the RMP. 

The importance of something as mundane as recordkeeping -- 
keeping an accurate and permanent record of each student's remedial 
status — should not be overlooked, either for RMP placement or for RMP 
evaluation. Overall, about 3/4 of our respondents reported that they 
keep a record of which RMP courses students are required (or 
recommended) to take (Table A17). However, where tests are not used 
for placement, keeping a record becomes atypical (Table A18). me 
proportion of institutions maintaining records is, not surprisingly, high 
(90%) when placement into RMP courses is mandatory (Table A19). 

Just how all the characteristics of the RMP's and their colleges may 
affect t..eir evaluative efforts is far from obvious. This issue will be 
addressed in the following chapter, which begins with an analysis of 
which colleges evaluate their programs. 
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Table 17 

Percent of RMP's in Which Placement 1 5 Mandatory 

(n=553) 
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Is Placement Mandatory? 

Yes 41% 

No 37% 

It Depends on the RMP Course 6% 

It Depends on the Student's Major 22% 

It Depends on the Student's Preparation 13% 

Other 7% 
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RMP EVALUATIONS 
Colleges Which Evaluate Their RMP's 

Nearly half (49%) of our responding institutions with remedial math 
programs indicated that their programs had been evaluated (Table A20). 
By contract, a national survey conducted a decade ago (Baldwin et al. 
19^5) reported that only 14% of responding institutions had ever had their 
RMP's "formally 11 evaluated* While the difference between the two figures 
may stem in part from sample error or from variations in the wording of 
questions, it seems likely that the more than tripling of the rate of 
reported RMP evaluation is due, at least in part, to increased interest 
and activity in program evaluation. 

Which colleges evaluated their RMP's? In an effort to address this 
question, we considered a rumber of (1 ) institution? characteristics, 
(2) RMP characteristics and (3) RMP practices, and whether these 
parameters are related to the tendency to evaluate. The first category 
included such institutional variables as type, size, and college-level math 
requirement, while the second included RMP goals, size, and age. In the 
third category, we examined such practices as whether tests are used for 
placement, whether placement is mandatory, whether there is an adequate 
number of sections, and whether student records are maintained. 

An analysis of the more than 500 programs in our sample discloses 
that most of the variables which we considered are not related to the 
decision to evaluate (Tables A20, A21, A22 and A23). It is mainly the 
placement practices which impinge on the likelihood that a program will be 
evaluated. These practices are: the use of testing for placement, 
mandatory placement, and the maintenance of placement records. In fact, 
of the institutions which follow all three practices, the rate of evaluation 
rises to 65%. 

About half of the institutions which use tests for placement in RMP 
courses have evaluated their RMP's (52%) in contrast to 22% of those not 
using tests (Table A24). Institutions at which placement into RMP 
courses is mandatory are almost twice as likely to have evaluated their 
RMP ! s ('63%) as those at which placement into the courses is optional (35%) 
(Table A25). In addition, institutions which keep a student record of 
required or recommended RMP courses are more than twice as likely to 
have evaluated their RMP's (59%) as institutions which do not keep such 
records (23%) (Table A26). 

Why should colleges which impose and keep a record of test-based 
student placement* tend to evaluate their RMP's? There are several 
possible explanations. For one tnng, where placement is mandatory, 
colleges will probably have more at stake in their RM'^'s, and so may be 
particularly anxious to determine and improve their effectiveness. 
Another factor ! s that the availability of placement (pretest) scores makes 
it relatively easy to measure the growth of student achievement for 
purposes of program evaluate, with minimal disruption to normal 
practice. Finally, the college's decision to allocate sufficient resources to 
test incoming students, to develop recordkeeping systems for storing RMP 
placement data, and to create mechanisms at registration or elsewhere to 
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enforce the RMP assignment may reflect a high degree of institutional 
commitment to the basic skills effort. The RMP evaluation is simply 
another aspect of that commitment. 

We also found that institutions at which all students are required to 
take college-level math courses are much more likely (57%) to have 
evaluated their RMP's than those at which no students have such a 
requirement (20%). However, the small number of institutions in the 
second category makes it difficult to establish a statistically significant 
relationship (Table A27). 

Description of the Evaluations 

In this section we get to the heart of our investigation — a 
description of the more than 200 RMP evaluations carried out in the 
responding colleges. Contrary to what we expected with the current 
emphasis on accountability, the evaluations which institutions described as 
their "most thorough effort conducted to date 11 tended to be internal 
projects. Less than half of the evaluations (40%) were carried out as the 
result of requests from outside the program (Table 18). Moreover, about 
two thirds of these "outside" requests still originated from within the 
institutions — 41% were from the department and an additional 25% from 
the administration (Table 18). Furthermore, the vast majority of the 
evaluations (86%) were conducted without the benefit of special funds 
allocated for the purpose. At those fortunate institutions for which funds 
were allocated (16%), the most common sources of funding were the 
administration (46%) and governmental units and agencies (37%) (Table 
19). Finally, less than 10% of the institutions used an outside evaluator 
(Table 20) . The low reliance on external evaluators is very likely a 
result of the lack of funds being allocated for the evaluation, if not a 
fear of too critical an assessment. Because so few institutions relied on 
external evaluators, there is little point in our comparing evaluations 
conducted internally with those conducted by an outside evaluator. 

Although the evaluations tended to be internal, this does not 
necessarily imply that they were modest in scope. While some 60% only 
used one approach (Table A28) and a similar proportion used only one 
type of student comparison (Table A29), the typical evaluation addressed 
several distinct questions (Table A30) and Involved several kinds of data 
(Table A31). 

Many options confront the RMP evaluator in considering how to 
evaluate a program. For example, in choosing an overall approach, the 
evaluator may focus on hard empirical student achievement data, 
employing the toois of statistical analysis; or instead rely on the judgment 
and experience of knowledgeable persons reacting to the current state of 
the program. Other approaches are, of course, possible. The most 
common approach for evaluating RMP's among our respondents was to 
examine the effects of the program on student performance — these 
effects were examined by 88% of all evaluating institutions and 80% of 
those using only one approach. By contrast, about half of all of the 
evaluations (and less than 20% of the institutions which used only one 
approach) were process-oriented, focusing on an examination of RMP 
policies and practices (Tables 21 and A32). 
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Table 18 

Percent of Evaluating Institutions at Which 
the Evaluation Was the Result of 
an Outside Request 

(n=267) 



Yes 40% 
No 60% 



Total 10 o% 



If Yes, the Source of the Request 
(n=96) 



Department 41% 

Administration 25% 

Governmental Unit 19% 

Private External Agency 0% 

Other 16% 

Total 100% 
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Table 19 



Percent of Evaluating Institutions at Which 
Funds Were Allocated for the Evaluation 

(n=270) 



Yes 16% 
No 84% 



Total 100% 



If Yes, the Source of the Funding 
(n=41) 



Department 7% 

Administration 46% 

Governmental Unit 37% 

Private External Agency 5% 

Other 5% 

Total 100% 
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Table 20 



Percent of Evaluating Institutions Which 
Used an Outside Evaluator 

(n=272) 



Yes g% 
No 92% 



Total 100% 



Table 21 



Percent of Evaluating Institutions Which Followed 
Various Approaches in the Evaluation 



(n=254) 



Approach Followed 
Examination of Policies and 

Practices Followed jn the RMP 51% 

Analysis of the Effects of the 

RMP on Student Performance 88% 

Other 2% 
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An evaluation can address any number of questions — program 
effectiveness, program modification, placement effectiveness, 
cost-effectiveness, and so forth. While most of our evaluating institutions 
considered more than one question in the evaluation of their R'dP's, 
certain questions and patterns of questions were commonly taken up. 
More than nine out of ten of the evaluating institutions (94%) addressed 
the question of the effectiveness of RMP courses (Tables 22 and A33). 
In fact, over three-fourths of the evaluations dealing with only one 
question focused on this issue. The effectiveness of the system for 
placement into RMP courses was the next most common concern — 74% of 
the evaluating institutions and 19% of those addressing a single question 
focused their evaluation on this issue. Following these questions in 
importance were the appropriateness of the RMP curriculum (59%) and 
directions in which the RMP should be modified (56%). A few respondents 
concentrated on somewhat narrower concerns such as: the quality of RMP 
pedagogy, the qualifications of RMP faculty, the effectiveness of RMP 
textbooks, and whether the program actually operates as it is supposed 
to. 

Whether the evaluation was conducted as the result of an outside 
request seems to have influenced the selection of questions which the 
evaluation address3d. There was a tendency for evaluators acting 
without an external impulse to be more interested in improving the 
program — in formative evaluation; institutions at which there was no 
outside request for the evaluation were more likely (62%) to address the 
question of RMP modification than those at which there was an external 
impulse (46%) (Table A34). On the other hand, those evaluators with an 
external impulse tended to be more interested in summative evaluation — 
in assessing the overall worth of the program. 

The reported evaluations usually involved more than one kind of 
information — only 3% relied on a single kind. Not surprisingly, certain 
types of data were used more frequently than others. In particular, 
measures of student performance — RMP grades (81%), test scores (73%) 
and subsequent math grades (72%) were commonly employed (Table 23). 
One explanation for the heavy use of these types of data is the 
importance which is normally attached to them in student (as opposed to 
program) evaluation. On the other hand, the heavy reliance on these 
measures may reflect the relative ease with which they are collected 
rather than a particular belief in their importance. Following these types 
of information in frequency of use were: student opinions of the RMP 
(49%), college retention rates (44%), and RMP faculty opinions of the 
program (41%) . Other types of information were employed much less 
frequently. 

On the whole, the tendency to use a particular kind of information 
was not influenced by whether the evaluation was conducted as a result 
of an outside request, or whether funds were allocated for the purpose. 
There are, however, some notable exceptions. For example, RMP faculty 
opinions of the program were more likely to come up in an evaluation 
when there was an outside request (49%) than when there was not (36%) 
(Table A35). Perhaps RMP coordinators who conduct evaluations as a 
result of an outside request tend to fear the exposure of weaknesses and 
therefore rely on internal perceptions to bolster the strength of the 



ERIC 



O -32- 



Table 22 

Percent of Evaluating Institutions at Which the 
Evaluation Addressed Various Questions 

(n=263) 



Question Addressed 

Are the Courses Effective? 94% 

Is the Placement System Effective? 74% 

Is the Curriculum Appropriate? 59% 

Is the RMP Cost-effective? 27% 

How Should the RMP Be Modified? 56% 

Other 3% 
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Table 23 

Percent of Evaluating Institutions Which Used Various 
Kinds of Information in the Evaluation 

(n=267) 



Kinds of Information Used 

Grades, Passing Rates, etc. in Remedial Math Courses 81% 

Test Scores 73% 

Grades, Passing Rates, etc* in Subsequent Math Courses 72% 

Student Opinions of the RMP 49% 

College Retention Rates 44% 

RMP Faculty Opinions of the Program 41% 

Student Attitudes Toward Math 27% 
Grades, Passing Rates, etc- in Subsequent 

Science/Technical Courses 17% 

Non-RMP Faculty Opinions of the Program 17% 

College Graduation Rates 13% 

Judgment of Visiting Experts 8% 

Other 3% 
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program. Not surprisingly — given the cost of employing consultants — 
those institutions at which funds were allocated for the evaluation were 
more likely to rely on the judgment of visiting experts (29%) than were 
institutions at which funds were not allocated (4%) (Table A36). 

A comparison of RMP students before and after the RMP — the 
pretest-posttest design — was the design most commonly used in the 
evaluations (58%), followed by a comparison of students completing the 
RMP with those who were exempted (40%), a comparison of students 
completing the RMP with students needing but not taking it (24%), and a 
compansion of recent RMP completers of recent RMP completers with 
previous completers (Table 24). It is likely that colleges relied on these 
designs because of the relative ease of their implementation — particularly 
the first — in contrast to more difficult designs such as a comparison of 
local results with those at other colleges (3%). The significance of the 
three most popular designs is confirmed by the high proportion of 
evaluating institutions using a single comparison in their evaluation that 
used each — 61%, 22% and 10% respectively (Table A37). Very few 
colleges (7%) employed a design involving a comparison with a national 
norm — quite a different result from what one would get in the evaluation 
of reading programs, for example, where national standardized tests for 
placement are more common. 

In short, we found that the typical evaluation was conducted 
internally, without special funding, and was based on data which were 
more or less readily available. However, this description only telis part 
of the story. We went a step further and investigated the relationships 
among various components of the evaluation — the questions addressed, 
the approaches taken, the types of information employed, and the 
comparisons drawn. The resulting statistics are presented in Tables A38 
through A43. While some of the relationships are puzzling and in fact 
may only be statistical flukes, others are quite understandable. 

Among the latter is the following relationship: evaluations 
considering placement effectiveness were much more likely to compare 
students who completed the RMP with exempted students than those which 
did not consider placement effectiveness (45% vs. 24%) — clearly a 
strategy for investigating cut scores . Not surprisingly , institutions 
which examined the effects of the RMP on student performance tended to 
rely heavily on such measures of achievement as test scores (77% vs. 
42%), RMP grades (84% vs. 68%), grades in subsequent math courses (78% 
vs. 32%), and retention rates (48% vs. 19%). By contrast, those which 
examined policies and practices of the RMP were significantly more likely 
to use student opinions of the RMP (63% vs. 33%), student attitudes 
toward math (40% vs. 15%), opinions of RMP faculty (56% vs. 28%), and 
the judgment of visiting experts (13% vs. 2%). Comparisons of students 
before and after the RMP were more likely to use test scores than those 
evaluations eschewing this comparison (85% vs. 61%), no doubt because 
other measures of achievement are often not available for students prior 
to remediation. Also, institutions which compared students completing the 
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Table 24 



Percent of Evaluating Institutions Which Used Various 
Comparisons in the Evaluation 

(n=245) 



Comparison Used 

Students Before and After RMP 58% 

Students Completing RMP vs. Exempted Students 40% 

Students Completing RMP vs. Students Needing But Not Taking 24% 

Recent RMP Completers vs. Previous RMP Completers 10% 

Local RMP Completers vs. a National Population 7% 

Local RMP Completers vs. RMP Completers at Another College 3% 

Other 3% 

No Comparisons Were Made 17% 
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RMP with exempted students were more likely (85% vs. 69%) to rely on 
test scores (perhaps surprisingly), and also tended to use grades and 
passing rates in subsequent math courses (91% vs. 64%) ~ measures 
readily available for the exempted population. Probably for a similar 
reason, evaluations involving a comparison of students completing the RMP 
with those needing but not taking remedial math courses tended to rely 
on grades in subsequent math (86% vs. 71%) and science/technical courses 
(31% vs. 13%). 

We also investigated whether the nature of a college's evaluative 
activities was related in any way to specific characteristics of the RMP. 
Here we were less successful in teasing out patterns, although a couple 
of interesting, if tentative, relationships did emerge. We found that the 
oldest, RMP's were much more likely to adopt a historical perspective — to 
compare recent RMP completers with previous completers in their 
evaluations (Table A44). Also, colleges with an insufficient number of 
RMP sections to meet student demand were more likely to compare 
students completing the RMP with students needing but not taking 
remedial math courses, no doubt a reflection of their strategy to create a 
control group of "leftover" students without withholding remediation from 
students and exposing the evaluator to the charge of callousness (Table 
A /*5). However, in neither case was the relationship statistically 
significant, if only because of a skewed sample. 

Of course, the circumstances under which many of these evaluations 
were conducted may have been far from ideal, resulting in compromised 
designs or implementation. In some cases, decisions were no doubt made, 
if not forced, as a result of circumstances beyond the control of the 
evaluator. Therefore before drawing any conclusions about past 
evaluation trends, and prior to making any suggestions for future RMP 
evaluative work, we will turn our attention to the comments of our several 
hundred respondents who have gone through the evaluation process, to 
gain insight into what problems they faced, what results they achieved, 
and how they perceived the experience. 
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AFTER THE EVALUATION: REACTIONS AND CONSEQUENCES 



Before making the commitment to evaluate a remedial mathematics 
program, one should be convinced that the job of planning and carrying 
out an evaluation is worth the effort. An informal evaluation of a small , 
single-course RMP may only entail minimal effort; however, a 
comprehensive, longitudinal study of a multi-course, multi-campus 
program can be a huge undertaking. The experience of our respondents 
— the extent to which they were satisfied with their evaluations, the 
problems they encountered, the results the evaluations produced, and any 
advice which they would give to someone about to design or conduct an 
RMP evaluation — can provide us with some insight into the value and 
cost of the evaluation process. 

Satisfaction and Problems 

When asked whether they were satisfied with the most thorough 
evaluation conducted on their RMP to date, the majority of our 
respondents answered our closed-ended question in the affirmative: yes, 
they were satisfied both in terms of design and conduct (Tables 25, 26 
and A46) . On the other hand, evaluations in which no comparison of 
student outcomes was employed generally led to dissatisfaction. 
Respondents from institutions using an outside evaluator were more likely 
to be satisfied with the evaluation design and conduct than those without 
outside evaluators, however the differences are not statistically 
significant. In their open-ended comments, a majority of respondents 
indicated some reservations about the evaluation, suggesting that the 
typical respondent was generally but not completely satisfied with the 
evaluation. 

The most frequently voiced criticisms dealt with deficiencies of data 
employed in the evaluation. In the typical criticism, the data were 
characterized as subjective or superficial (Tables 27 and 28). A related 
objection was the lack of longitudinal or follow-up information, either in 
courses immediately following the RMP or in courses through to 
graduation. Obtaining such information requires a longer commitment 
than some evaluators were perhaps able or willing to make. 
Several respondents noted with disapproval the lack of other kinds of 
data as well — opinions from outside the department, multiple measures of 
achievement, etc. 

Another concern was the problem of access to or availability of data. 
Some criticisms dealt with information which was lost or never obtained. 
This information was of many types, including posttest scores of students 
who dropped out of the program or college, or who were never in the 
RMP; various kinds of data for which a satisfactory recordkeeping system 
had not been developed; or measures of post-RMP achievement for 
students who did not advance to a college-level math course immediately 
upon exiting from the program. Other criticisms related to the denial of 
access to data by instructors, departments or the administration. In 
short, underlying problems seemed to be poor planning , lack of 
cooperation, lack of resources, oi logistical difficulties. 
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Table 25 

Percent of Evaluating Institutions Satisfied 
with the Evaluation Design 

(n=245) 



Satisfied? 

Yes 74% 

No 27% 

Total 100% 



Table 26 

Percent of Evaluating Institutions Satisfied 
with the Conduct of the Evaluation 

(n=242) 



Satisfied? 

Yes 81% 
No 19% 



Total 100% 
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Table 27 

Percent of Evaluating Institutions with Various 
Reactions to the Evaluation Design 

(n=63) 



Reaction 

Too Superficial 33% 

Additional Data Needed 29% 

In Progress 14% 

Generally Adequate 13% 

Design Problems 11% 

Inadequate Resources 8% 



Table 28 

Percent of Evaluating Institutions with Various 
Reactions to the Conduct of the Evaluation 

(n=34) 



Reaction 

Too Superficial 29% 

Additional Data Needed 21% 

Generally Adequate 18% 
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Poor cooperation from various segments of the college community was 
cited in connection w ; th a number of other difficulties. One respondent 
noted that some course instructors disliked the evaluation because they 
considered it to be an evaluation of their performance. Several others 
pointed to indifference or hostility from the Remediation Center, non-RMP 
facu' y or the administration. 

Many of the respondents pinpointed particular design or measurement 
problems — problems no doubt related to the concern expressed earlier 
about the quality or quantity of evaluative data. Half a dozen 
respondents indicated that comparisons such as those described in the 
survey questionnaire should have been but were not made. A particular 
design complaint was the lack of a control group with which to compare 
the RMP "graduates," a failir.^ noted frequently in the literature (e.g., 
Piescc 1978). One important observation was that the success of students 
in RMP courses depends on a large number of variables, making it 
difficult to identify causal relationships. Several respondents expressed 
disapproval of trie tects used to obtain data. A number of individuals, 
while admitting deficiencies in their evaluation, attempted to absolve 
themselves of any liability, blaming instead the governmental agency or 
external unit which imposed on them design or time constraints. 

Another object of criticism was the nature or size of the student 
sample on which the evaluative &tudy was based. Some respondents 
complained that the sample employed was either too small or not 
sufficiently representative of the larger student population for the study 
to yield generalizable results. The basic problem here was probably a 
decision to be expedient in choosing to work with students on whom data 
were most easily obtained; a lack of resources to handle larger quantities 
of date may also have contributed to the difficulty. 

As to the particular problems which were encountered in conducting 
the evaluation, one of the most common was inadequate resources (Table 
29). The insufficiency of many different kinds of resources was cited, 
including funding, time, released time, staffing, computer support, 
secretarial help, technical expertise and general support. 

Some respondents had an unkind word for the individual or 
individuals responsible for the evaluation. One argued generally against 
outside evaluators: "People from outside the RMP program have very 
little understanding of the total program, bat they were in charge of the 
evaluation." Another wrote of the poor communication between the 
evaluators and the faculty involved in the RMP. 

Finally, there were a number of technical and logistical problems 
singled out as alfecting the evaluation. These were quite varied, and 
included: the short life of the program, getting placement testa given to 
students before they enter class, coordinating the administration of tests 
on two campuses, and computer keypunching. 

With all the obstacles which RMP evaluators seem to have faced, it's 
easy to forget that overwhelmingly, respondents reported being satisfied 
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Table 29 

Percent of Evaluating Institutions Encountering 
Various Problems in the Evaluation 

(n=92) 



Problem 

None 25% 

Inadequate Resources 19% 

Inadequate Data 15% 

Lack of Support 10% 

Inadequate Access to Data 8% 

Design Problems 6% 

Technical /Logistical Problems 5% 
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with their RMP evaluations. One of the reasons for this satisfaction may 
be the program improvements to which the evaluations led ~ a subject to 
which we now turn. 

Recommendations and Changes 

If RMP evaluations tend simply to "rubber stamp" dominant program 
practices or to suggest program changes which never get implemented, 
then the value of even the most minimal effort expended in the evaluation 
process would be questionable. However, the experience of our 
respondents indicates that neither of these depressing scenarios is 
typical. More often than not, thorough evaluations do include 
suggestions for modifying the RMP in one way or another, and these 
modifications generally seem to be implemented (Tables A47 and A48). In 
a phrase, evaluations can make a difference. 

The great variety of recommendations from their evaluative studies 
which our respondents reported reminds us of the large number of 
threads which are entwined in a remedial mathematics program. 
Untangling these threads and selecting directions for program change can 
challenge any RMP evaluator or administrator. That recommendations from 
different institutions are frequently at cross purposes with one another 
should not be seen as evidence of their invalidity. Rather this Brownian 
motion with its apparently random variation reflects the way in which 
educational research differs from educational evaluation. In research, the 
investigator attempts to identify educationally significant principles which 
hold in the broadest range of possible settings. An evaluation, however, 
focuses on a particular program in a particular institution with a 
particular mission, particular resources, and particular constraints at a 
given point in time. No wonder then that at different institutions, 
evaKations lead to conflicting recommendations. 

Table 30 summarizes the changes in RMP's most frequently 
recommended in program evaluations, while Table 31 shows the changes 
actually implemented as a result of evaluations. It is clear from these 
tables that in more than half the cases, recommendations in evaluative 
studies led to program change. 

More recommendations dealt with placement procedure than any other 
area. The suggested placemen* changes were of many types. Quite a 
few advocated changing the placement exam; often by developing an 
in-house exam. Other recommendations had to do with adjusting cut 
scores on the placement exatt, and, in one case, changing the length of 
the exam. A number of evaluations proposed that the placement exam be 
administered to broader categories of students. Several comments dealt 
with moving toward mandatory placement into RMP courses, while one 
advocated moving in the opposite direction toward voluntary enrollment, 
and still others with the imposition of a college-wide competency standard. 

Some recommendations involved suggested changes in RMP pedagogy, 
often a shift from self-paced instruction toward the more structured, 
conventional group-paced instruction. From our small sample of 
comments, there is the suggestion that whatever the advantages of 



Table 30 



Percent of Evaluating Institutions with Various 
Recommended Changes Resulting 
from the Evaluation 

(n=117) 



Recommended Change 




Change Placement Criteria 


23% 


Change Instructional Method 


13% 


Redesign Course(s) 


12% 


Change Course Content 


10% 


Change Exit Criteria 


8% 


Create Math Lab 


6% 


Change Course Sequence 


5% 


Add Courses 


4% 


Separate Courses 


3% 


Combine Courses 


2% 




Table 31 

Percent of Evaluating Institutions with 
Various Changes Implemented as a 
Result of the Evaluation 

(n=117) 



Implemented Change 

Change Instructional Method 16% 

Change Placement Criteria 16% 

Change Course Content 12% 

Redesign Course 11% 

Change Exit Criteria 8% 

Change Curricular Materials 7% 

Separate Courses 5% 
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self-paced instruction, the associated disadvantages — procrastination, 
special staffinn, grading and registration procedures among them — are 
increasingly beir.g viewed as carrying greater weight. 

Many evaluations contained proposals for modifying RMP goals, 
curricula or courses. Typical were recommendations to clarify or change 
course objectives in the light of new circumstances; to add or take away 
courses in arithmetic, algebra, trigonometry or plane geometry; and in 
one way or another to extend, re-order or re-organize course topics. 
Some suggestions had to do with changing the RMP course structure, 
sometimes by modularization, other times by altering the course sequence 
or by merging separating courses. Other suggestions dealt with the 
addition of courses for special student populations (e.g., students who 
only need a brief review). 

A number of recommendations advocated redesigning courses in 
special ways — making them more practical and less theoretical, 
upgrading the content, integrating remedial and non-remedial content in a 
new course, creating two-week workshops at the end of the semester for 
students who did not pass the posttest, increasing the number of contact 
hours which RMP courses meet, or changing the schedule of RMP courses 
in one way or another. 

Some suggested changes focused on improving support services. 
Typical among them were recommendations that more tutoring be provided; 
that the amount of lab facilities be increased; that computer-assisted 
instruction be offered; that a Math Lab be established, dropped or 
restructured; and that a student assistant for each class be provided. 
Several recommendations supported changing the textbook or seeking a 
text with prescribed features. 

Many respondents reported recommendations concerning RMP exit 
criteria, grading practices, or the granting of credit for RMP courses. A 
common recommendation was to upgrade the content in RMP courses and 
also to raise the standards for successful completion. Representative 
comments were: "Stop offering remedial math (disguised as business 
math) for college-credit," and "No longer allow basic algebra to meet a 
graduation requirement in general education. (However, still let it count 
towards the 124 hours required for graduation.)" A related 
recommendation was to standardize exit criteria — particularly when many 
sections of a course are taught by adjunct faculty. Other proposals 
supported changing the grading system to pass/fail (S/U), or modifying 
policy concerning the granting of the grade of incomplete — often a 
controversial issue under self-pacing. 

Some suggestions had to do v vith teaching and testing strategies, as 
well as with student- teacher relations. Several recommended that the 
RMP courses provide additional practice in solving exercises of a practical 
nature. Several of the recommendations on testing supported the 
adoption or discontinuation of mastery testing ( retesting-until-mastery) . 
Others caiied for improving communication between students and faculty 
by holding conferences regularly, or by having faculty periodically write 
follow-up letters and make telephone calls to students. Specific 
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pedagogical recommendations included placing more emphasis on explaining 
the meaning of new mathematical terms — especially pertinent to students 
who are weak in reading or study skills — and having each student set 
his or her own course goals. 

Organization, attendance, recordkeeping, funding and location were 
dealt with in a number of recommended changes. Several involved taking 
the RMP out of the mathematics department ~ its traditional home ~ and 
moving it to another administrative unit within or even outside the 
college. Other administrative recommendations included restricting 
attendance — a measure sometimes adopted when programs have low 
passing rates — computerizing RMP student recordkeeping, and moving 
RMP classes to the Learning Center. Still others related to improving the 
program's cost-effectiveness; one idea here was to "replace the 
state-funded remedial algebra course with a student-funded workshop." 

Some proposals advocated changes in RMP staffing, staff development 
and resources. Several dealt with establishing procedures for training 
teaching assistants, tutors or other support staff, sometimes through 
workshops. One idea was to require first-semester instructors to attend 
weekly seminars. Other recommendations were to change a particular 
instructor, hire additional staff, offer more sections, or reduce average 
class size. 

Only one respondent reported an evaluation which explicitly 
recommended that his or her RMP be continued. The absence of this 
particular recommendation from other evaluations is quite striking. This 
absence not only suggests that the continuation of RMP's seems to be 
almost universally taken for granted, but also that the underlying goals 
of RMP evaluation are as much formative (how can we improve the 
program?) as they are summative (how good is the program?). This 
pattern has important implications for RMP evaluation design, a theme to 
which we shall turn in the concluding section of this monograph. 

Advice 

On our questionnaire, respondents were invited to give advice of 
any sort to someone about to design or conduct an RMP evaluation. The 
resulting comments are categorized in Table 32. They are extremely 
varied. In many cases, they simply voice issues which had surfaced in 
the earlier section of the questionnaire dealing with evaluation problems 
— sufficiency of data, data comparisons, and sufficiency of resources, 
for example. However, these responses also raise other concerns not 
previously addressed in our analysis of survey results. 

For one thing, quite a few respondents emphasized what should but 
may not be obvious — the importance of planning ahead in an RMP 
evaluation. Several comments urged the would-be evaluator to consult 
with others who have been involved with evaluations of the sort desired, 
to read the professional literature for design alternatives, and to 
otherwise "do your homework." One of many reasons for planning an 
evaluation ahead is to avoid losing irretrievable data, such as pretest 
scores and attitudinal measures prior to remediation. One piece of advice 
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Table 32 



Percent of Evaluating Institutions Giving Various Types of 
Advice on the Design or Conduct of an RMP Evaluation 

(n=83) 



Advice 

General Advice 20% 

Importance of Using an Outside Evaluator 13% 

Importance of Identifying the Program Needs 10% 

Advice on Statistical Analysis and Design 9% 

Importance of Getting Faculty and Administrative Support 8% 

Importance of Seeking Certain Characteristics in the Evaluator 7% 

Importance of Looking at Passing Rates in Non-remedial Courses 6% 
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urged colleges to design the RMP with evaluation in mind, 
institutionalizing pre- and posttests, questionnaires, etc., in order to 
conduct program evaluations on a regular basis with minimal disruption to 
normal practice. 

Some of the suggestions cealt with the results and recommendations 
which the final evaluation report should contain, and how they ought to 
be arrived at. One respondent cautioned against jumping to conclusions 
before the analysis is completed. A second advised the evaluator to "be 
prepared for depressing results, 11 a caution which might better have 
focused on the avoidance of unreasonable expectocions. A third, 
apparently well trained by his college's administration, wrote: "Regardless 
of the conclusions drawn at the end of the evaluation, don't (!!) 
recommend more resources, personnel or money — any changes 
recommended should use existing personnel resources and money," 
Another respondent emphasized the importance of keeping the final 
evaluation report simple and clear in order to reduce the chances that it 
winds up dustladen on an administrator's shelf. 

A number of respondents expressed their views on who should 
conduct an RMP evaluation. Suggestions included: a respected member 
of the mathematics department, faculty who teach RMP courses, faculty 
who teach college-level math and science courses, and research-oriented 
mathematicians. Many respondents suggested that outside evaluators be 
employed to conduct the evaluation, presumably because of their 
perceived or actual impartiality, their commitment to carry out the 
project, and their technical and statistical expertise. One person noted 
that whoever designed the evaluation, it was imperative that "someone 
carry it through to the end." 

Some advice urged the clarification or specification of RMP goals and 
needs. Typical was: "Clearly identify and state course objectives, then 
evaluate with respect to the specified objectives" — a point of view which 
would be opposed by evaluators preferring goal-free assessment. One 
respondent wrote of the importance of RMP and non-RMP math courses 
having goais which articulate; another, disdainful of affective 
considerations, felt that "the only meaningful criterion is the success of 
students in achieving academic goals." 

A number of respondents argued for drawing various comparisons in 
the evaluation — one of the issues at the heart of any evaluation. Some 
of the designs advocated were the pretest-posttest comparison ar d the 
cross-college comparison, that is, comparing the effectiveness of the local 
RMP with that of an RMP at a comparable college. One respondent wrote 
of the importance in evaluation of controlling such variables as teacher, 
time of day, and student background differences. Several respondents 
asked for # rather than gave, advice on the question of evaluative 
comparisons, such as "I wish I knew how to design an evaluation so that 
I could offer some constructive suggestions." 

Some advice dealt with securing different kinds or sufficient 
quantities of data for the RMP evaluation. Most common were comments 
stressing the importance of longitudinal data. Typical was: "Be sure to 
include a follow-up on students after they exit from the RMP -- how do 



they do in subsequent math courses/ 1 or again, "Our main concern is 
what happens in subsequent courses (we found that only students who 
make A's in remedial courses are good risks for regular math courses)." 
Some of the other suggestions were to gather multiple measures of 
achievement in order to have a well-rounded assessment. One respondent 
noted the importance of being carefui to consider the background of RMP 
students when comparing them with others. The value of accurate 
recordkeeping was emphasized in several comments, both in RMP and 
post-RMP courses. 

A number of respondents commented on tests and other instruments 
for gathering data for the evaluation. The most common advice was to 
employ a standardized test in the evaluation. One respondent gave her 
opinion on measuring attitudes and opinions: "Use well-planned, 
face-to-face interviews — they are worth a hundred questionnaires; 
interviewees should be chosen at random." 

Many comments were political and pragmatic, focusing on the 
desirability and ways of securing faculty and administrative support for 
the evaluation. One such comment suggested that the evaluator "explain 
why the evaluation is needed, how it will benefit the development of the 
program, and present efficient and well thought out procedures for 
accomplishing it." A less optimistic respondent noted: "Don't expect 
help from the administration or money either!" 

All in all, the concerns raised by our respondents in the advice 
which they gave and in their comments on past evaluations suggest that 
the responsibility of evaluating an RMP must not be taken casually. The 
message is clear: unless care is exercised in avoiding a host of potential 
difficulties, the evaluation's payoff may be minimal. On the other hand, 
with sufficient attention to avoiding these difficulties — with careful 
planning, a qualified evaluation team, the cooperation of colleagues and 
administrators, an appropriate choice of data and data comparisons, and 
sufficient resources of many kinds — an RMP evaluation can be designed 
and carried out which can shape the future of the RMP for the better. 
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CONCLUSIONS AND RECOMMENDATIONS 



In the course of this investigation, we have examined the 
responses from some 700 institutions and seen the extraordinary scope 
of the remediation challenge with which the postsecondary mathematics 
community is confronted. We have learned that the overwhelming 
majority of co. leges, at least in our sample, offer remedial math 
programs and that on the average colleges with RMP's judge the 
algebraic or arithmetic preparation of 1 out of every 3 entering 
students to be inadequate. Moreover, we have seen how RMP's have 
grown and become firmly entrenched until today their continued 
existence has come to be taken for granted. 

This monograph has dealt with many aspects of RMP's — the 
context in which they operate, their goals, their curricula, and their 
enrollment patterns, to name but a few. However, our primary concern 
has been to document the kinds and frequency of RMP evaluations 
which colleges have conducted to date. To carry out this task, we 
have identified a number of methodological characteristics which taken 
together constitute an evaluation profile. Most notable among these 
characteristics have been: the general evaluative approach taken, the 
questions addressed in the evaluation, the kinds of data collected and 
analyzed, the types of student comparisons (designs) employed, the 
nature of the resources and personnel that were available for the 
evaluation, and finally the problems which came up in the course of 
implementation. 

In this concluding chapter, we draw from our survey findings 
some suggestions for future RMP evaluation and research. 



(1 ) We recommend that colleges periodically undertake rigorous 
evaluations of their RMP's^ ~ 

While the notion that colleges should regularly conduct a rigorous 
assessment of their RMP's may seem trite, we do not make this 
recommendation lightly. Quite the contrary, for our survey provides 
evidence of serious difficulties commonly associated with RMP 
evaluations. However, of greater significance is the fact that the 
hundreds of program evaluations on which our respondents have 
reported more often than not led to significant improvements in 
RMP's — improvements covering such pivotal program elements as 
placement criteria, instructional method, and course design or content. 
We believe that many of these changes would not have been effected, 
let alone considered, had the evaluative studies not been conducted, 
and that they constitute a tangible payoff for the time, effort, and 
resources invested in the evaluation. 
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(2) We recommend that evaluations should assess the appropriateness 
of the RMP curriculum, identify the program's strengths and 
weaknesses, and suggest improvements in the program . 

Not surprisingly, the two issues most commonly addressed in the 
hundreds of surveyed RMP evaluations were: (i) the effectiveness of 
the system of placing students into RMP courses and (ii) the 
effectiveness of these courses. Clearly, both issues are important and 
should be dealt with in any study. 

However, it seems of dubious value to assess placement and course 
effectiveness before first establishing the appropriateness of the course 
content. This concern for curricular validity is vital during those 
periods of time when program goals or circumstances are shifting 
significantly. We believe that technological and professional forces 
already at work may well contribute to such shifts in the near future. 
For one thing, the full curricular consequences of the calculator and 
the computer have yet to be felt in RMP's. Then again, various trends 
in college and high school curricula — for example, greater emphasis on 
problem solving and the applications approach in the high school, the 
introduction of discrete mathematics into the college curriculum, and 
shifting statistics and logic downward into the high school — make it 
essential that evaluative studies ascertain whether RMP curricula are 
meshing with both secondary and postsecondary courses. 

We'd also like to advocate that future RMP studies stress formative 
evaluation at least as much as summative evaluation. Unlike a 
summative study which assesses the overall worth of a program, a 
formative evaluation identifies the strengths and weaknesses of a 
program, and recommends ways in which the program should be 
modified. 

While RMP evaluations frequently involve the gathering of opinions 
on ways to improve the program from various constituencies, and some 
evaluation strategies have formative as well as summative value, 
nevertheless, the typical RMP evaluation is unquestionably designed so 
as to gather evidence on the overall worth of the existing program. In 
our opinion, RMP evaluations should also include empirical studies 
specifically geared toward program improvement. Such studies might 
investigate any of a number of important aspects of the program: 
whether students who repeat RMP courses or ESL students succeed as 
well as other students, whether tutorial or computer-assisted instruction 
is more effective, whether reading courses should be taken prior to or 
concurrent with math courses, whether arithmetic and algebra should be 
taught in separate courses or together, whether adding a math lab hour 
improves passing rates, and so forth. Exploratory investigations 
dealing with these and other program issues can provide valuable clues 
to identify desirable changes. 
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[II J"*™ 0 ™™^ th3t RMP evalu ations f ocus on data which measure 
the extent to which program goals are achlevecL 

,niioI h | e P !* ima l goal . of most RMP ' S ^ to prepare students to master 
Sim?t.l mathe f mat '«. Consequently, RMP evaluations, whether 

S^^? < sr at ! v *', ,ho i d focus on measures ° f student success in 

subsequent college-level math courses. However, we found that the 
most commonly employed type of evaluative data were not grades in 

SSrth "nf C ° UrSeS **? i radeS eamed in the RMP itself ' M ore than a 
fourth of our reported evaluations failed to include any measure of 

ach.evement n subsequent college-level math courses TW Hack of 

crmcsm in th a t in H f0 t rmati T m JX be the basis of the frequently voiced 
criticism that data employed in evaluations were superficial. 



(4 i. We recomme nd that summative RMP evaluating compare the 
achievement ,n tollow -up courses of students who have S R MP 

uiLl ?LT .^ St L den tS WhU " eeaed bUt dld not receive remediation . -in^ 
wi th siuaents who were initially exempted from remediitfoTT ' 

As noted previously, most evaluations in our survey used one or 

Amono tt ™ (desi 9 n > *> assess program Effectiveness 

Among the comparisons were- the pretes t-posttest desiqn the 

Se 0 sTa"n 9r °an P d de tt qn ' thL^medi^ e f^emp^ 

design, and the remediated-unremediated desmn" A striking trend 

emerges when we look a I the frequency of designs, a trend whfch says 
as much about human nature as about program evaluation: the easier a 
"LarofTvnL 10 ^ T' ih \ m0re fre ^ntly it is conducted This 
thev are in th^ PMP Plai r n , S h Why com P arin 9 students before and after 
they are in the RMP (the pretest-posttest design) is used so 
frequently: it is by far the simplest comparison to implement. 

Unfortunately, the pretest-posttest comparison -has several 
Hecht iQ^^^p 0 " 6 ° f ^ WhiCh might be considered fatal (Akst and 
,ftPr ail V'm *° r one i hm 9' !t near| V always yields positive results; 

even under "tX" V^f'" 9 - h ° Ur5 a Week ifl remediai curses 

and scorf £ W ?7, ° f w clr ^ um u stances ^e bound to learn something 

i there is ! «*in h -° f the C ° UrSe than at tne start ' Secondly 
if there is a gain, the design sets no standard of acceptability with 
regard to the size of that gain: how much is enough? A^her 

Prettf from' * that the . gains Calculated b " subfrac?ng 

pretest from posttest scores may be only loosely related to the 

program's effectiveness; the gains could have resulted from learning or 
reviewing outside the course, from the statistical phenomenon of 
regression toward the mean, from students simply trying harder on the 
fmal exam than on the placement test, or from 7 a 9 host of other 
uncontrolled and confounding variables. Finally, the desgn ignores 
improved performance in follow-up math classes - generally the 
program goal. In other words, it substitutes learning fbr 9 retentten and 
transfer as measures of success. retention ana 
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Just as the pretest-posttest design asks the question Do students 
know more at the end of remediation than at the beginning, so in the 
norm-group comparison, the issue is: How does the improvement of 
students in the RMP compare to the improvement of a comparable, 
national group of students on whom test norms have been published? 
The cross-program design contrasts the gains registered by students in 
the local RMP with those in another program; it asks the question: 
Which program is doing a better job with comparable students? The 
fact that both the norm-group and the cross-program designs were used 
infrequently — for example, the cross-program design was found in 
only 3% of surveyed evaluations — is almost certainly attributable to 
associated pragmatic difficulties. These range from finding a suitable 
standardized test with the right content, norm-group, price, 
administration time, and a sufficient number of equivalent forms, to 
finding comparable RMP's willing to participate in a politically sensitive 
experiment. 

Leaving aside the weak pretest-postest design and the impractical 
norm-group and cross-program designs, this leaves us with two 
comparisons, each of which seems both valid (in the sense of generating 
cogent evidence) and workable. The remediated-exempted design 
compares students who have passed RMP courses with others who were 
originally exempted from these courses. The stringent question here 
is: Do students who have gone through the RMP subsequently perform 
as well as students who originally had not needed remediation? The 
relevance of this question to typical program goals is obvious. Finally, 
there is the classic control group comparison — the 
remediated-unremediated design — in which students who have gone 
through the program are contrasted with students who were diagnosed 
as needing remediation bv't did not enroll in RMP courses. This design 
asks: If students who passed remedial courses had not had the benefit 
of remediation, would they have turned out as well? 

Neither design is as easy to implement as the pretest-posttest 
comparison. In carrying out the remediated-unremediated design, for 
example, it may be difficult to find initially comparable students outside 
the RMP either because of mandatory placement or because of the 
s e|f-selection bias. Furthermore, both designs will somehow have to 
take into account another group of students, namely those who start 
out in the RMP but never pass. Despite such vexations, we are 
encouraged by the fact that significant percentages of our surveyed 
evaluations used one or the other of these designs, hopefully finding 
ways around difficulties in their implementation. 

The choice of an evaluation design — like the choice of type of 
data — will have a major impact on the evaluation's results. It may 
"stack the deck" to a considerable extent in advance of the study's 
actual implementation. Thus it is essential, if an evaluation is to be 
more than a self-serving exercise, that the design and data combine 
qualities of intellectual integrity and practicability. 
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(5) We believe that there is a need to develop a consensus on what 
constitutes v alid methodology in an kMP evaluation . 

The 1930's were the "shaking out" period for the accounting 
profession in America. That decade saw 5 search for basic tenets 
which would eventually become today's Generally Accepted Accountinq 
Prmci; ,es (CAAP). A 1936 editorial in one of the leading accounting 
journals stated the problem: y 

After a quarter of a century and more of active 
discussion and experimentation in this country, 
many of the simplest and most fundamental problems 
of accounting remain without an accepted solution. 
There is still r .o authoritative statement of essential 
principles available on which accounting records and 
statements r:ay be based. Public accountants ... 
have been nsked to certify to the correctness and 
adequacy of accountinq statements when no 
satisfactory criteria of correctness and adequacy 

have been agreed to What is wrong with 

accounting as the public sees it? Could it be the 
fact that there is stiil no broad authoritative code 
of accounting principles? 

RMP evaluators in the 1980's confront a similar challenge. The 
diversity of methods employed in th* nrme of assessment provides little 
confidence in their validity and may simply reflect a lack of standards. 
Frequently employed but questionable evaluation practices such as the 
use of RMP grades as measures of program success or the use of the 
pretest-posttest design without controlling extraneous variables h=>ve 
the effect of exaggerating the chances of programs taking home report 
cards more favorable than they deserve. Such strategies, whether 
adopted because of limited resources, lack of knowledge, expediency or 
even self-interest, bring evaluation into disrepute. Clearly, those of 

^AA n n 0 , Ve< ?., ' n RMP ' S and their evaluati <>n need to develop our own 
MJAAP. While agreement on a single evaluation model may not be 
feasible if only because of significant variation in local parameters, the 
adoption of broad principles of RMP evaluation such as the few 
recommended above could be a guide for future practice, and also serve 
as a preemptive strike igainst budget-cutting state legislatures. 

As a means of accomplishing this goal, the major national 
organizations such as AMATYC or the National association of 
Developmental Educators (NADE) would do well to reach agreement on 
generally accepted principles in RMP evaluation, or at the leasl on what 
constitutes the key questions and nncerns which should be addressed 
in an evaluation. Furthermore, the force of these organizations behind 
a statement of minimal resources for adequately evaluating a program 
could be helpful to future faculty evaluates. With luck, specific 
models for evaluation of RMP's might be identified as worthy of the 
imprimatur of these organizations. Another route to achieve a similar 
goal would be to reach a consensus at a gathering of leading program 
evaluators and RMP coordinators from around the nation, preferably 
with support from a civic-minded, private foundation or a governmental 
agency such as the Fund for the Improvement of Postsecondarv 
Education (FIPSE). y 
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(6) We recommend that RMP coordinators become familiar with the 
principles and practices of program evaluation , ~~ ~~ 

Since less than 1 out of every 10 evaluations reported in our 
survey involved an outside evaluator, it is clear that college faculty 
and staff — often the remedial coordinators — are evaluating their own 
programs. It is up to them to wrestle with the large number of 
approach, design and data issues which confront any evaluator. 

As we have already -bserved, the stand taken with respect to 
these issues will to a m^or extent determine the outcomes of the 
evaluation. It is critical that such decisions be based on a sound 
knowledge of the prus and cons of alternative evaluation strategies. To 
foster this understanding, we'd like to see graduate schools of 
education encourage prospective college math faculty ~ particularly 
those headed for public two-year institutions — to take courses in 
program evaluation. Faculty who have assumed the post of RMP 
coordinator most particularly need to be conversant with the RMP 
evaluation literature. 



(7) We recommend that additional research be conducted on patterns of 
RMP evaluation . 

Many of our findings have confirmed the hypothesis that the 
characteristics of both a college znd its RMP help to shape it« 
evaluative activity. To recall just ore of many examples, we've learned 
that the likelihood of evaluating an RMP seems to be related to an 
institution's overall commitment to mathematics as measured by such 
indicators as whether the college requires all students to take a course 
in college-level mathematics, or all students found in need to be 
assigned to the appropriate RMP course. 

The discovery of such relationships depends on the characteristics 
under investigation. Other researchers may wish to select variables 
different from ours and to explore connections among these new college, 
program and evaluation profiles. One of a number of major RMP 
characteristics which we have ignored is the mode of instruction 
dominant in the program; a hypothesis which we believe worth testing 
is that RMP's with unconventional instructional practices are most likely 
to be evaluated, or to be evaluated in some special way. In addition, 
an interesting evaluative trait which we might have included on the 
questionnaire had space permitted is whether the RMP assessment was 
based on a value-added or a mastery-learning philosophy. That is, was 
program success measured by the number of students completing RMP 
courses whose achievement increased by a certain amount or by the 
number of these students who attained a pre-specified level of 
competency? Such distinctions in the groundrules of a study can 
radically alter findings. 
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e *nlnrpH ny p ^ C| , uestions related to RMP evaluation need to be 

122- t'hP Zr anC6 / ? hat d6SignS and data are bein 9 "sed to 
assess the efficacy of placement procedures? And what of the 

S S t^2T e H n V S ,° f RMP ' S - Wha? kinds of evidence are be ng 
?hev nut in det ! rmine L f C ? Me 9 es 9 et more ° ut of their programs than 
they put in, m terms of cold, financial investment? 

interel^tJL™ 1 1C 5 + ° f ^ uestions of both practical and theoretical 
interest beckoning to future researchers. 
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THE QUESTIONNAIRE 



tA$ City Vrmrnty of New 
bukmctioneJ Resource Center 




vtrt 



MatfeesMtkal Assertion 
C Two Tear CoHeies 



April 22, 1983 

Dear Colleague: 

//i i resent years, the effectiveness of basic mathematics skills oroarams 

co^tectfv© experience some guidelines for future program evalua^ 
m instructional Resource Center of the City Unh^ofNeXand 
the American Mathematical Association of Two Year Colleoes an Ttoint^ 

SSSSif * UrV * y 01 W°*<*es used to evaluate rTmMa, 
mathematics programs. 

We are requesting that you either fill out this questionnaire or forward it 
%Z HE*!? indivkiual onyour campus, s/ress/n^ /m^ce 
o^o/ecf. The questionnaire should be returned to us as Son at 

prlvZZ'm^ 01 th ° co"' 1 **"'*"* of the information you 

data. All respondents will be sent a copy of the completed repen. * * 
Thank you for your participation in this important project. 

Cordially, 




Geoffrey Akst 

Chain Evaluation Subcommittee 
of the Developmental Mathematics 
Curriculum Committee, AMATYC 

Marie Jean Lederman 

University Dean, Academic Affairs 

Director, Instructional Resource Center, CUNY 
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A Questionnaire on the Evaluation of Remedial Mathematics Programs 



This questionnaire is designed to gather information about the evaluation of remedial mathematics 
programs For purposes of this questionnaire, one or more remedial mathematics courses constitute 
a remedial mathematics program (RMP). Throughout, the term evaluation referstoa 2erM&of* 
° * T' PW ram ™ le »* na ' cessment activities. Please answer the questions Mow by 
filling in the blanks or where Indicated, by placing a check mark beside the 4oroSTs^ 
Ptease ignore the italicized numbers; they are for processing responses. Thank y6u for your cooTra- 
Hon in completing the questionnaire and returning it as soon as possible. 



Part I. College Characteristics 

[7] 1. Typo of Institution: 

1 □ Public two-year college 

2 □ Private two-year cofege 

3 □ Public four-year college 

4 □ Private four-year cofege 

5 □ Other (please specify) _ 



18-12] 



.2. Approximately how many undergraduate 
students attend your college? 



77VT/>r- 1 Approximately how many freshmen enter your 
P3-16] college each fall? 



[17] 



[18-19] 



4. Are students at your college required to take 
one or more college-level mathematics 
courses to graduate? 

1 □ All students are required to take one or 

more cofege-ievel mathematics courses. 

2 □ Students in certain majors are required to 

take one or more cofege-tevel mathe- 
matics courses. 

3 □ All students must meet distribution 

requirements toward which college-level 
mathematics courses count 

4 □ Students in certain majors must meet cto- 

trfcrjtion requirements toward which 
coHege-tevel mathematics courses count 

5 □ No students are required to faka cotege- 

tevel mathematics courses. 

6 □ Other (please specify) . 



[20] 5. Does your college offer any remedial 
mathematics courses? 
1 □ Yes 2 □ No 



Iff the answer to Question 5 is yes, please proceed to Part II; 
[otherwise proceed to Part IV. 



Part IL Characteristics off Your Remedial 
Mathematics Program 

6. in what year dW.your RMP begin? 

[21-22] 



[23] 
[24] 
[25] 



[26-27] 



[28] 
[29] 
[30] 

[31] 
[32] 
[33] 



y. Which of the following are the major goals of 
your RMP? (check all that apply) 

□ To prepare students for subsequent 
mathematics courses 

O To prepare students for subsequent 
science/technical courses 

□ To prepare students to meet a college, 
university, or system-wide standard of 
competency 

□ Other (please spscify) 



8. Which areas of mathematics are covered in 
the RMP courses at your college? (check all 
that apply) 

□ Arithmetic 

□ Elementary Algebra 

□ Intermsdfete Algebra 

□ Trigonometry 

□ Plane Geometry 

□ Pre-calculus 

□ Other (please specify) 



[34-35] 

8. How many courses are offered in your RMP? 

[3&37] 

10. What kind(s) of information do you use to 
place students into your RMP? (check ail that 
apply) 

□ High school record 

□ Scores on an admissions test (e.g. , SAT, 
ACT) also used for placement 

□ Scores oh atocaBy developed placement 
test 

□ Scores on a commercially available 
placement test 

□ Other (please specify) 

11. Approximately what percentage of the 
college's incoming freshman class is judged 
to be in need of one or more RMP courses? 



[38] 
139] 

[401 
[41] 

[41-43] 



[4445] 



erJc 



12. On the average, how many students are 

[46-50] enrolled in the RMP at any given point to time 
during the regular academic year? 
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[51] 
[52] 
[S3] 

m 

[55] 
[5&57] 



13. Is placement into an RMP course mandatory 
tor students who are found to be inadequately 
prepared in mathematics? (check aR that apply) 

□ Yes 

□ No 

□ ft depends on the RMP course. 

□ It depends on the student's major. 

□ It depends on the tevel of the student's 

mathematics preparation. 

□ Other (pease specify) m 



[58] 14. Are there usualy enough sections of RMP 
courses to accommodate student need? 
1 □ Yes 2 □ No 

[59] 15. Is a record kept for each student indicating 
which RMP courses) the student is required 
or recommended to take? 
1 □ Yes 2 □ No 

[60] 16. Have any efforts been made to evaiuate the 
RMP at your college? 
1 □ Yes 2 □ No 



if the answer to Question 16 is yes, please proceed to Part 
Hi; otherwise proceed to Part IV. 



Part III. Evaluation of Your Remedial 
Mathematics Program 

Piease answer the remaining questions with respect to the 
most thorough evaluation of your RMP conducted to date. 

[61] 17. Was the evaluation conducted as a result of a 
request from outside your program? 
1 □ Yes 2 □ No 

[62] If yes, who made the request? 

1 □ The department 

2 □ The coflege or university administration 

3 □ A governmental unit 

4 □ A private external agency which funds 

part or all of the RMP 

5 □ Other (please specify) _ 



[63-64] 



[65] 18. Were there funds allocated for the specific 
purpose of conducting an evaluation of the 
RMP? 

1 □ Yes 2 □ No 
[66] If yes, what was the source of the funds? 

1 □ The department 

2 □ The coflege or university administration 

3 □ A governmental unit 

4 □ A private external agency which funds 

part or al! of the RMP 

5 □ Other (piease specify)_ 



[7071] 
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[69] 19. Did any evaluator(s) from outside your college 
or university participate In evaluating the 
RMP? 

1 □ Yes 2 □ No 



[72] 



If outside ©valuators participated in the 
evaluation, piease describe the procedures 
used to chocs* them: 



If yes. piease Bst the nam@(8)and affiHattonfc) 
of the evaiuator(8): 

Name 



Affiliation 



1. 



[73] 
[74] 

[75] 
[76] 
[77] 

[7$-7§] 



20. Which of the Mowing questions dti the 
evaluation address? (check al that appty) 

□ Are the RMP courses effective? 

□ Is the system for placement into RMP 
courses effective? 

' □ Is the RMP curriculum appropriate? 

□ te the RMP cost-effective? 

□ How should the RMP be modffied? 

□ Other (piease specify) 



21. What WndB of Information were used in the 
evakjatton? (check a H that apply) 
[80] □ Test scores 
[81] □ Grades, passing rates, etc. in RMP 
courses 

[82] □ Grades, passing rates, etc. in subsequent 

mathematics courses 
[83] □ Grades, passing rates, etc, in subsequent 

science/technical courses 
[84] □ Retention rates (in the college) 
[85] □ Graduation rates 
[86] □ Student opinions of the RMP as 

measured by questionnaires, interviews, 

etc. 

[87] □ student attitudes toward mathematics as 
measured by questionnaires, interviews, 
etc. 

[38] □ Opinions of RMP faculty of the program as 
measured by questionnaires, interviews, 
etc. 

[89] □ Opinions of nonflMP faculty of the program 
as measured by questionnaires, inter- 
views, etc. 

[80] □ The judgment of one or more visiting 
experts 

pf.&] n Other (please specify) , 



22. Which of the following approaches were used 
in the evaluation? (check all that apply) 

□ An examination of the policies and 
practices followed in the RMP 

□ An analysis of the effects of the RMP on 
student performance (l.e., grades, reten- 
tion rates, graduation rates, etc.) 

□ Other (please specify) 



[93] 
[94] 

[95*98] 
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23b if the evaluation considered student perfor- 
mance, (fid it involve any of me totoving 
comparisons? (check al that app»V>; 

□ RMP students before and after $MP 
courses 

□ Students who completed an RMP course 
with students exempted from the course 

□ Students who completed an RMP course 
with students who needed, but cSd not 
take the course 

□ Students who recently completed an 
RMP course with students who com- 
pleted the course previously 

□ Students who completed an RMP course 
at your college with students who com- 
pleted en RMP course at another cdege 

□ Students who completed an RMP course 
at your college with a national population 
used to norm a standardized test 

□ Other (please specif)) 



D7] 
P0Q1 

P01] 
[1OM04] 

[105] □ No comparisons were made 

[106] 21 Does your RMP provide support services (e.g. 
tutoring, computer-assisted Instruction)? 
1 □ Yes 2 □ No 

nnfimi tf please describe row. if at ail, 
1 lu '' 1ua J these services have been evaluated: 



[U7] 2»o D£ the evaluation lead to any recommend 
Hons for changes in the RMP? 
iDYes 20No 

If yes. please describe the recommendations: 

[11&119] 



[1*0] 29. Were there any changes Implemented in the 
RMP as a result of the evaluation? 
1 □ Yes 2 □ No 

» yes, pteaoe describe the changes: 

[121-122] 



[12&124] 



30. What advice would you give to someone 
about to design or conduct an RMP 
evaluation? 



[109] 25, Are you satisfied with the way the RMP 
evaluation was designed? 
1 □ Yes 2 □ No 

[110111] Oomm8n * 



Please attach any available descriptions of evaluation 
designs, or research and evaluation reports on your RMP 
The Information will be kept in confidence. 



Part IV. Respondent 

Name 



[112] 2a Are you satisfies with the way the RMP 
evaluation was conducted? 
1 □ Yes 2 □ No 

[mTu] CmM 



27. .What particular pnAlems were encountered in 

[1 15-1 16} conducting the evaluation? 



Title. 



College 



[125} 



[126-127] 



Department 

College Address. 



Telephono Number 



Pteesontum Ux 



Geoffrey Akct 

Instructional Resource Center 

The Ctty University of New Yort 

535&a0thSL 

Hew York, New York 10021 
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SUPPLEMENTARY TABLES 



Table A1 

Region: A Comparison of the Sample 
and the Population 

(n=648) 



Region Sample Population 

West/ Southwest 22% 23% 

Southeast 24% 23% 

Great Lakes/ Plains 30% 26% 

North-Atlantic 24% 28% 



<d -67- 
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Total 100% 100% 

P(X2)=.08 
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Table A2 



ERIC 



Type of Institution: A Comparison of 
the Sample and the Population 

(n=628) 





Sample 


Population 


Public 2-year 


32% 


33% 


Private 2-year 


7% 


11% 


Public 4-year 


26% 


16% 


Private 4-year 


35% 


'41% 


Total 


100% 


100% 


P(X')=.00 
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Table A3 



College Enrollment: A Comparison of 
the Sample and the Population 

(n=616) 



College Enrollment 


Sample 


Population 


Up to 1000 


28% 


34% 


1001-5000 


43% 


43% 


5000+ 


29% 


23% 


Total 


100% 


100% 


P(X J )=.00 







Table A4 

Percent of Institutions Offering Remedial Math 
Courses by Whe'her All or No Students Are 
Required to Take College-level Math Courses 



Required Students 

All Students 86% 
(n=209) 

No Students 65% 
(n=23) 
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Table A5 



Percent of RMP's with Various Coals 
by Number of Coals 



Goal 



Number 
of 

Coals 
1 

(n=226) 



(n=198) 



(n=118) 
(n=11) 



To Prepare For: 



Subsequent 
Math 
Courses 

77% 



99% 
100% 
100% 

90% 



Subsequent 
Science/ 
Technical 
Courses 

3% 



69% 
99% 
100% 
49% 



A Standard of 
Competency 

17% 



29% 
92% 
100% 
39% 



Total 
(n=553) 



Sting'ofgoals. 0 " ** qUestlonna,re ln ^ App.„dfx «* * 
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Table A6 



Percent of RMP's with Various Goals 
by Type of Institution 



Goal 



Public 2-year 
(n=195) 

Private 2-year 
(n=27) 

Public 4-year 
(n=147) 

Private 4-year 
(n=162) 

Total 
(n=531) 



To Prepare For: 

Subsequent 
Subsequent Science/ A Standard 
Technical 



Math 
Courses 

94% 



89% 



91% 



85% 



90% 



of 

Courses Competen cy 
70% 39% 



37% 



42% 



35% 



50% 



37% 



39% 



38% 



39% 



,03 



.00 



.99 
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Table A7 



Median Number of Students Enrolled in RMP 
Courses by the Year That the RMP Began 



The Year the RMP Began Median 

Before 1960 400 
(n=18) 

1960-1964 460 
(n-26) 

1965-1969 338 
(n=64) 

1970-197^ 200 
(n=112) 

1975-1979 105 
(n=152) 

1980 and Later 35 
(n=74) 

Total 125 
(n=538) 
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Table A8 

Percent of RMP's with Various Enrollments 
(n=538) 



Enrollment 

Up to 50 31% 



51-300 
301 + 



39% 
30% 



Total 10 0% 

Median=1 25 



Table A9 

Percent of RMP's Which Provide 
Support Services 

(n=264) 



Provide Support Servicas 88% 
Don't Provide Services 13% 
Total 100% 



-73- 



82 



Table A10 

Percent of RMP's Which Provide Support 
Services by the Percent of Freshmen 
in Need of Remedial Math Courses 



Percent of 
Freshmen 
in Need of 
RMP Courses 

1-20% 
(n=77) 

21-49% 
(n=82) 

50-99% 
(n=85) 



Percen- 
Providing 
Support 
Services 

81% 



84% 



95% 



Total 87% 
(n=244) 

P(X J )=.01 
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Table An 



Percent of RMP's with an Insufficient Number 
of Sections by Type of Institution 



Public 2-year 16% 
(n=195) 

Private 2-year 7% 
(n=5) 

Public 4-year 18% 
(n=146) 

Private 4-year 6% 
(n=160) 

Total 13% 
(n=528) 

P(X J )=.00 



Table A12 

Percent of Institutions with RMP's with Selected Ranges^of 
Freshmen in Need of Remedial Math Courses 

(n=508) 



Percent of Freshmen 

in Need of Percent of 

Remedial Math Courses Institutions 

1-20% 33% 

21-49% 31% 

50-100% 36% 

Total 100% 

Median=35% 
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Table A13 



Percent of Institutions with RMP's with at Least 50% 
of the Students in Need of Remedial Math Courses 



Public 2-year 48% 
(n=176) 

Private 2-year 44% 
(n=25) 

Public 4-year 37% 
(n=136) 

Private 4-year 21% 
(n=155) 



Total 
(n=492) 

P(X*)=.00 



36% 
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Table A14 



Percent of RMP's Which Use 
Various Types of Placement Information 
by Type of Institution 







Tvdg of Placement 


2 n "formation 






HS 
Record 


Admissions 
Test 


Local 
Test 


Commercial 
Test 


Public 2-year 
(n=192) 


50% 


31% 


52% 


34% 


Private 2-year 
(n=27) 


41% 


26% 


59% 


30% 


Public 4-year 
(n=147) 


47% 


51% 


61% 


19% 


Private 4-year 
(n=161) 


49% 


53% 


44% 


37% 


Total 
(n=527) 


48% 


43% 


52% 


30% 


P(X^) 


.83 


.00 


.02 


.00 
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Table A15 



Percent of RMP's Which 
Use Various Numbers of Types 
of Placement Information 

(n=549) 



Number of Types 

of Placement Percent of 

Informat.on Used Institutions 

1 42% 

2 32% 

3 24% 

* 2% 

5+ 0% 

Total mn£ 



Median=2 



a See Question 10 on the questionnaire in the 
Appendix for a listing of types of placement 
information. 
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Table A16 



Percent of RMP's in Which Placement Is 
Mandatory for All Students by Type of Institution 



Public 2-year 31a 

(n=194) 31 * 

Private 2-year 67ft 

(n=27) 67% 

Public 1-year 43% 
(n=148) 

Private 4-year noa 
(n=162) 



Total 
(n=531) 

P(X*)=.00 



41% 
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Table A17 

Percent of RMP's Which Keep a Student 
Record of Required or Recommended 
Courses by Type of Institution 



Public 2-year 66% 
(n=192) 

Private 2-year 85% 
(n=27) 

Fublic 4-year 74% 
(n=144) 

Private 4-year 79% 
(n=158) 

Total 73% 
(n=521) 

P(X') = .02 



ERIC 
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Table A18 



ERIC 



Percent of RMP's Which Keep a Student Record of 
Required or Recommended Courses by Whether 
Tests Are Used for Placement 



Percent 
Keeping a 
Record 

Tests Used 77% 
(n=493) 

Tests Not Used 32% 
(n=44) 

Total 73% 
(n=537) 

P(X 2 )=.00 



Table A19 

Percent of RMP's Which Keep a Student Record 
of Required or Recommended Courses by 
Whether Placement Is Mandatory 



Percent 
Keeping a 
Record 

Placement Mandatory 90% 
(n=222) 

Placement Not Mandatory 61% 
(n=319) 
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Table A20 



Percent of Institutions with RMP's Which Evaluated 
Their RMP by Type of Institution 



9 
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Public 2-year 43% 
(n=195) 

Private 2-year 59% 
(n=27) 

Public 4-year 
(n=148) 

Private 4-year 49% 
(n=163) 

Total 4 9% 
(n=533) 

P(X 2 )=.15 



191 



Table A21 
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Percent of Institutions with RMP's Which Evaluated 
Their RMP by the Year That the RMP Began 



The Year That the RMP Began 

3efore 1960 57% 
(n=18) 

1960-1964 52% 
(n=27) 

196b-1969 47% 
(n=66) 

1970-1974 52% 
(n=115) 

1975-1979 c.10 
(n=153) 



i o 



1980 and later 40% 
(n=80) 

Total 49% 
(n=459) 

P(X 2 )=.58 
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Table A22 

Percent of Institutions with RMP f s Which 
Evaluated Their RMP by Coals 



Coal 



To Prepare For: 



An Not an 
RMP RMP 
Coal Coal P(X 2 ) 



Subsequent Math Courses 49% 47% .87 

(n=500) (n=53) 

Subsequent Science/ 

Technical Courses 51% 48% 



(n=272) (n=281) 
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.53 



A Standard of Competency 54% 46% .13 

(n=215) (n=338) 



Table A23 



Percent of Institutions with RMP's Which 
Evaluated Their RMP by College Enrollment 



Percent 
Which 

College Enrollment Evaluated 

Up to 1000 45% 
(n=136) 

100:-5000 51% 
(n=238) 

5001+ 50% 
(n=171) 

Total 49% 
(n=545) 

P(X 2 )=.48 
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Table A24 



Percent of Institutions with RMP's Which 
Evaluated Their RMP by Whether a 
Test Is Used for Placement 



Test Used 52% 
(n=503) 

Test Not Used 22% 
(n=46) 

Total 49% 
(n=549) 

P(X 2 )=.00 



Table A25 

Percent of Institutions with RMP's Which 
Evaluated Their RMP by Whether 
Placement Is Mandatory 



Mandatory 63% 
(n=228) 

Not Mandatory 
(n=205) 
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Table A26 

Percent of Institutions with RMP's Which 
Evaluated Their RMP by Whether Student 
Records Are Kept of Required or 
Recommended Courses 



Record Kept 59% 
(n=394) 

Record Not Kept 23% 
(n=149) 

Total 49% 
(n=543) 

P(X 2 )=.00 



Table A27 

Percent of Institutions Which Evaluated Their RMP 
by Whether All or No Students are Required 
to Take College-level Math Courses 



All Students 57% 
(n=179) 

No Students 20% 
(n=14) 
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Table A28 



Percent of Evaluating Institutions Using Various 
Numbers of Approaches in the Evaluation 

(n=273) 



Number of Approaches 

1 60% 

2 39% 

3 1% 
Total 100% 

Median=1 



See Question 22 on the questionnaire in the 
Appendix for a listing of approaches. 
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Table A29 



Percent of Evaluating Institutions Using 
Various Numbers of Comparisons 
in the Evaluation 

(n=245) 



ERIC 



Number of Comparisons 

1 58% 

2 27% 

3 10% 

4 4% 

5 1% 



Total 100% 

Median=1 



See Question 23 on the questionnaire in the 
Appendix for a listing of comparisons (designs). 
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Table A30 



Percent of Evaluating Institutions 
Addressing Various Numbers of 
Questions in the Evaluation 

(n=263) 



Number of Questions 

1 

2 

3 

4 

5 

6 

Total 



12% 
22% 
24% 
26% 
16% 
0% 
100% 



Median=3 



See Question 20 on the questionnaire in the 
Appendix for a listing of evaluative questions. 



Table A31 



Percent of Evaluating Institutions Using Various 
Numbers of Kinds of Information 3 
in the Evaluation 

(n=267) 



Number of Kinds of Information 



1 


3% 


2 


14% 


3 


17% 


4 


22% 


5 


20% 


6 


10% 


7 


6% 


8 


6% 


9 


1% 


10 


1% 


Total 


100% 



Median=4 



See Question 21 on the questionnaire in the 
Appendix for a listing of kinds of information, 
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Table A32 



Percent of Evaluating Institutions Which Followed 
Various Single Approaches in the Evaluation 

(n=152) 



Single Approach Followed 
Examination of Policies and Practices 

Followed in the RMP 19% 
Analysis of the Effects of the RMP 

on Student Performance 80% 
Other 1% 
Total 100% 



Table A33 

Percent of Evaluating Institutions Which Addressed 
Various Single Questions in the Evaluation 

(n=31) 



Single Question Addressed 

Are the Courses Effective? 77% 

Is the Placement System Effective? 19% 

Is the Curriculum , .ppropriate? 0% 

Is the RMP Cost-effective? 0% 

How Should the RMP Be Modified? 3% 

Other 0% 

Total 100% 



Table A34 
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Percent of Evaluating Institutions Which Addressed Various 
Questions in the Evaluation by Whether the Evaluation 
Was the Result of an Outside Request 



No 

Outside Outside 

Request Request Total 

Question (n=104) (n=156) (n=260) P(XM 

Are the Courses Effective? 95% 9h% 94% .79 

Is the Placement System 

Effective? 83% 69% 74% .02 
Is the Curriculum 

Appropriate? 61% 58% 59% .82 

Is the RMP Cost-effective? 32% 24% 27% .20 
How Should the RMP Be 

Modified? 46% 62% 55% .02 
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Table A35 



Percent of Evaluating Institutions Which Used Various Kinds of 
Information in the Evaluation by Whether the Evaluation 
Was the Result of an Outside Request 



Kinds of Information 


Outside 
Request 
(n=107) 


Outside 
Request 
(n=157) 


Total 




Test Scores 


69% 


75% 


72% 


.41 


Grades, Passing Rates, etc. 
in Remedial Math Courses 


79% 


82% 


80% 


.65 


Grades, Passing Rates, etc. 
in Subsequent Math Courses 


70% 


72% 


71% 


.85 


Grades, Passing Rates, etc. 
in Subsequent Science/ 
Technical Courses 


20% 


16% 


17% 


.54 


College Retention Rates 


51% 


40% 


44% 


.10 


College Graduation Rates 


10% 


14% 


13% 


.48 


Student Opinion of the RMP 


49% 


48% 


48% 


.96 


Student Attitudes Toward 
Math 


31% 


24% 


27% 


.24 


RMP Faculty Opinions 
of the Program 


49% 


36% 


41% 


.05 


Non-RMP Faculty Opinions 
of the Program 


21% 


14% 


17% 


.22 


Judgment of Visiting Experts 


11% 


5% 


8% 


.11 
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Table A36 

Percent of Evaluating Institutions Which Used Various 
Kinds of Information in the Evaluation by Whether 
Funds Were Allocated for the Evaluation 



Funds 



Kinds of Information 


Funds 
Allocated 
(n=42) 


Not 
Allocated 
(n=225) 


Total 
(n=267) 


P(X2 


Test Scores 


79% 


72% 


73% 


.46 


Grades, Passing Rates, etc. 
in Remedial Math Courses 


81% 


80% 


81% 


1 .00 


Grades, Passing Rates, etc. 
in Subsequent Math Courses 


67% 


72% 


72% 


.57 


Grades, Passing Rates, etc. 
in Subsequent Science/ 
Technical Courses 


21% 


16% 


17% 


.57 


College Retention Rates 


55% 


42% 


44% 


17 


College Graduation Rates 


14% 


12% 


13% 


.94 


Student Opinion of the RMP 


55% 


48% 


49% 


.49 


Student Attitudes Toward 
Math 


31% 


26% 


27% 


.66 


RMP Faculty Opinions 
of the Program 


45% 


40% 


41% 


.64 


Non-RMP Faculty Opinions 
of the Program 


26% 


15% 


17% 


.11 


Judgment of Visiting Experts 


29% 


4% 


8% 


.00 
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Table A37 



Percent of Evaluating Institutions Which Used Various 
Single Comparisons in the Evaluation 

(n=102) a 



Single Comparison Used 

Students Before and After RMP 61% 

Students Completing RMP vs. Exempted Students 22% 

Students Completing RMP vs. Students Needing But Not Taking 10% 

Recant RMP Completers vs. Previous RMP Completers 2% 

Local RMP Completers vs. RMP Completers at Another College 2% 

Local RMP Completers vs. a National Population 2% 

Other 2 % 

Total 100 % 



Evaluating institutions which did not use student comparisons in the 



evaluation are excluded from this analysis. 
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Table A38 

Approaches Used in the Evaluation Which Are Significantly 
Related to Questions Asked in the Evaluation 



Question 

1) Are the RMP Courses 
Effective? 

2) !s the System for 
Placement into RMP 
Courses Effective? 

3) Is the RMP Curriculum 
Appropriate? 



4) Is the RMP Cost- 
effective? 



Approach 



Examination of Policies 
and Practices Followed 
in the RMP 

Exarrvnation of Policies 
and Practices Followed 
in the RMP 



Direction 
of Rela- 
tionship P(X 2 ) 



.00 



.00 



5) How Should the RMP 
Be Modified? 



Examination of Policies 
and Practices Followed 
in the RM^ 



,00 



A sign indicates that evaiuating institutions which addressed the 
question were more likely to use the indicated kind of approach; a "-" 
sign indicates that evaluating institutions which addressed the question 
were less likely to use the indicated kind of approach* 
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Table A39 



Comparisons Used in the Evaluation Which Are Significantly 
Related to Approaches Used in the Evaluation 



Approach 

Examination of Policies 
and Practices Followed 
in the RMP 



Analysis of the Effects 
of the RMP on Student 
Performance 



Comparison 

Students Completing RMP 
vs. Exempted Students 

Recent RMP Completers vs. 
Previous RMP Completers 

Students Completing RMP 
vs. Exempted Students 
No Comparisons Made 



Direction 
of Rela- 
tionship^ P(X 2 ) 



+ 
+ 



.05 
.00 



.02 
.00 



A "+" sign indicates that evaluating institutions which used the approach 
were more likely to use the indicated comparison; a "-" sign indicates 
that evaluating institutions which used the approach were less likely to 
use the indicated comparison. 
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Table A40 



Comparisons Used in the Evaluation Which Are Significantly 
Related to Questions Asked in the Evaluation - 



Question 

1) Are the RMP Courses 
Effective? 

2) Is the System for 
Placement into RMP 
Courses Effective? 

3) Is the RMP Curriculum 
Appropriate? 

4) Is the RMP Cost- 
effective? 



5) How Should the RMP 
Be Modified? 





Direction 






ot rela- 




- 

Comparison 


tionship 


PI f V 1 


- 

otuaents completing RMP 


- 


- 


vb • uxemp tea o iuaen is 


T 


.01 


rxecen l r\/vir completers vs • 






PrPY/tnilC RMP f nmnlotor c 

n cviuuo iMvir v^urupieierb 


T 


.U4 


Students Completing RMP 






vs. Students Needing 






But Not Taking Remedial 






Math Courses 


+ 


.02 


Recent Completers vs. 






Previous RMP Completers 


+ 


.01 


Students Before and After 






RMP 


+ 


.02 


Students Completing RMP 






vs. Exempted Students 


+ 


.02 


Recent RMP Completers vs. 






Previous RMP Completers 


+ 


.00 


No Comparisons Were Made 




.02 



A "+" sign indicates that evaluating institutions which addressed the 
.question were more likely to use the indicated kind of comparison; a "-" 
sign indicates that evaluating institutions which addressed the question 
were less likely to use the indicated kind of comparison. 
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Table A41 



Kinds of Information Used in the Evaluation Which Are Significantly Related 
to Approaches Used in the Evaluation 



Approach 

Examination of Policies 
and Practices Followed 
in the RMP 



Analysis of the Effects 
of the RMP on Student 
Performance 



Kinds of Information 

Grades, Passing Rates, etc. 

in Subsequent Math 

Courses 
Student Opinions of the RMP 
Student Attitudes Toward 

Math 

RMP Faculty Opinions of the 

Program 
Judgment of One or More 

Visiting Experts 

Test Scores 

Grades, Passing Rates, etc. 

in Remedial Math Courses 
Grades, Passing Rates, etc. 

in Subsequent Math 

Courses 
College Retention Rates 



Direction 
of Rela- 



tionship P(X 2 ) 



+ 
+ 
+ 

T 
+ 
+ 



+ 
+ 



.05 
.00 

.00 

.00 

.00 

.00 

.05 



.00 
.00 



A ,, + M sign indicates that evaluating institutions which used the approach 
were more likely to use the ir dicated kind of information; a sign 
indicates that evaluating institutions which used the approach were less 
likely to use the indicated kind of information. 
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Table A42 

Kinds of Information Used in the Evaluation Which Are Significantly Related 

to Questions Asked in the Evaluation 



Question 

1) Are the RMP Courses 
Effective? 



2) Is the System for 
Placement into RMP 
Courses Effective? 



3) Is the RMP Curriculum 
Appropriate? 



4) Is the RMP Cost- 
effective? 



Direction 



rvinub or intormation 


_ r r"> _ 1 

of Rela- 




^..a 
tionship 


P(X 2 


oraaes , rassing Kates, etc. 






in Subsequent Math 






Courses 


+ 


.00 


or dueb , rassing Kates, etc. 






hi rxcnicuiai (viatn courses 


+ 


.00 


kj races, rassing Rates, etc. 






iii ouusequent /viatn 






f^ni 1 rcoc 


+ 


.00 


r\ivir rdcui xy upi nions ot tne 






r 1 uy 1 alii 


+ 


.02 


Colleae RptAntinn Potoc 

vui l^vj^ IXC LCI 1 L IU 1 1 |\d LCd 




.05 


Test Scores 


+ 


.02 


College Retention Rates 


+ 


.03 


Student Opinions of the RMP 


+ 


.03 


RMP Faculty Opinions of the 






Program 


+ 


.03 


Ncn-RMP Faculty Opinions 






of the Program 


+ 


.03 


College Retention Rates 


+ 


.04 


College Graduation Rates 


+ 


.00 


Student Opinions of the RMP 


+ 


.03 


Student Attitudes Toward 






Math 


+ 


.04 


RMP Faculty Opinions of the 






Program 


+ 


.00 


Non-RMP Faculty Opinions 






of the Program 


+ 


.00 
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Table A42 
(continued) 



Question 

5) How Should the RMP 
Be Modified? 



Direction 
of Rela- 



Kinds of Information 


tionship 3 


P(X*) 


Test Scores 


+ 


.02 


Grades, Passing Rates, etc. 






in Remedial Math Courses 


+ 


.05 


Student Opinions of the RMP 


+ 


.01 


Student Attitudes Toward 






Math 


+ 


.04 


RMP Faculty Opinions of the 






Program 


+ 


.00 


Non-RMP Faculty Opinions 






of the Program 


+ 


.02 



a 

A "+" sign indicates that evaluating institutions which addressed the 
question were more likely to use the indicated kind of information; a "- 
sign indicates that evaluating institutions which addressed the question 
were less likely to use the indicated kind of information. 
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Table A43 



Kinds of Information Used in the Evaluation Which Are Significantly Related 
to Comparisons Used in the Evaluation 



Comparison 

Students Before vs. After 
RMP 



Kinds of Information 
Test Scores 



Direction 
of Rela- 
tionship^ P(X 2 ) 

+ .00 



Students Completing RMP 
vs. Exempted Students 



Students Completing RMP 
vs. Students Needing 
But Not Taking Remedial 
Math Courses 



Recent RMP Completers 
vs. Previous RMP 
Completers 



Local RMP Completers vs. 
a National Population 

Local RMP Completers vs. 
Completers at Another 
College 



Test Scores 

Grades, Passing Rates, etc. 
in Subsequent Math 
Courses 

Grades, Passing Rates, etc. 

in Subsequent Math 

Courses 
Grades, Passing Rates, etc. 

in Subsequent Science/ 

Technical Courses 
RMP Faculty Opinions of the 

Program 

Test Scores 
Graduation Rates 
Student Opinions of the RMP 
RMP Faculty Opinions of the 
Program 

Judgment of One or More 
Visiting Experts 

RMP Faculty Opinions of the 
Program 



+ 

+ 
+ 
+ 



.01 
.00 

.03 

.00 

.00 

.01 
.01 
.03 

.00 
.00 
.00 



T 



A ,, +" sign indicates that evaluating institutions which used the 
comparison were more likely to use the indicated kind of information; a 

sign indicates that evaluating institutions which used the comparison 
were less likely to use the indicated kind of information. 
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Ta~le A44 

Percent of Evaluating Institutions Which Compared 
Recent RMP Completers with Previous RMP 
Completers by the Year the RMP Began 



The Year the RMP Began 

Before 1960 
(n=9) 

1960-1964 
(n=13) 

1965-1969 
(n=28) 

1970-1974 
(n=50) 

1975-1979 
(n=72) 

1980 and later 
(n=31) 



Total 9% 
(n=203) 

P(X2)=.60 



22% 
8% 
4% 
12% 
10% 
7% 
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Table M5 



ERIC 



Percent of Evaluating Institutions Which Compared 
Students Completing the RMP with Students Needing 
But Not Taking RMP Courses by Whether There Was a 
Sufficient Number of Sections of RMP Courses 



Used the 

The Number of Sections Was: Comparison 

Sufficient 22% 
(n=215) 

Insufficient 32% 
(n=28) 

Total 24% 
(n=243) 

P(X2)=.36 
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Table A46 



Percent of Evaluating Institutions Which Were Satisfied with the Design of 
the Evaluation by Whether Various Comparisons Were Used 



Comparison 
Not Used 



Comparison n 

Students Before and 

After RMP 95 

Students Completing RMP vs. 
Exempted Students 131 

Students Completing RMP vs. 

Student Needing But Not 

Taking Remedial Math Courses 172 

Recent RMP Completers vs. 
Previous RMP Completers 202 

Local RMP Completers vs. 
Completers at Another College 220 

Local RMP Completers vs. 

a National Population 209 

No Comparisons Were Made 189 



Percent 
Satisfied 



68% 



66% 

73% 

73% 

74% 

74% 
80% 



Comparison 
Used 

Percent 
n Satisfied P(X 2 ) 



132 
96 

55 
25 



18 

38 



78% 
84% 

78% 

84% 

86% 

78% 
45% 



.14 
.00 

.53 

.33 

.78 

.92 
.00 
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Table A47 



Percent of Evaluating Institutions at Which 
Evaluations Led to Recommendations for Change 

(n=230) 



Recommendations for Change 56% 
No Recommendations for Change nn% 
Total 10 q% 



Table A48 



Percent of Evaluating Institutions at Which 
Recommended RMP Changes Were Implemented as a 
Result of the Evaluation 

(n=230) 



Recommended RMP Changes 
Were Implemented 55% 

Recommended RMP Changes 
Were Not Implemented 45% 

Total 100% 
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